I have a test project that I want to automate on a test server. For now, I have installed the Visual Studio agents (Test Controller, Test Agent and Test Lab) on my local machine, as I hope to try it out here before touching anything on the test server.
The problem I'm having is that I can't find adequate documentation on automating tests using the Visual Studio agent tools. I have successfully set up and configured the test agent and test controller on my local machine, but I'm not sure where I go from here. The test controller is connected to Team Foundation Server. I have a test project contained within TFS - I want to run the tests within it.
I tried launching the VS test manager and attempted to find a way to run tests from there, but I was unsuccessful.
How do I go about doing this? (in simple steps)
My previous experience is with NUnit, and launching automated tests was as simple is launching the command line NUnit program, providing it with the path of the test project and a few other parameters.
Once the Visual Studio agents are set up and configured correctly, it's simply a matter of running tests against a .testsettings file which is connected to the test controller.
When the tests are run (either via the Visual Studio GUI or the mstest.exe command-line program), the active testsettings file will run the tests on any test agent which is connected to the test controller.
So in terms of automating tests, it's just a matter of specifying the correct testsettings file as one of the command-line switches to the MSTest.exe.
A testsetting file can either be created within the Visual Studio solution or alternatively they can be created within Microsoft Test Manager (automated tests can also be scheduled using the MS Test Manager command-line program, tcm.exe).
To select an active testsetting file to use when running tests from Visual Studio 2010, select the 'Test' menu on the toolbar, then 'Select Active Test Settings'.
Related
I've got some CodedUI tests I'd like to run on a remote machine from visual studio. Now it used to be that you'd install a test controller somewhere, a test agent on the box you want to run on, get them talking, then use a test settings file that was pointed to that controller. I was going through this and installed agents for 2015 update 3 on my run box. I noticed that there is no controller configuration option. After some googling I found this: https://vstfsalm.wordpress.com/2015/08/18/tfs-2015-test-agent-has-not-been-configured-run-the-test-agent-configuration-tool-to-configure-the-test-agent/. Very cool. So I go in to tfs to set up a machine group. Great. The part I don't get is: how can I now tell VS to run at that group? Test settings files require a controller when they are set to remote run. Do I have to downgrade the agents back to 2013? Does anyone know if Microsoft just didn't implement this work style in 2015 (on purpose or otherwise) or if it just isn't implemented yet?
To run the code UI test via vNext build on TFS update3. First make sure the code UI test can be executed successfully in your Visual Studio on your remote machine locally. Also don't forget to configure the build agent to interact with desktop.
Then you just need to add Deploy TestAgent on RemoteTestMachine task and run Functional test. When you queue a build, it will deploy your code ui test solution to the drop folder on the build server machine. Then use MSTEST to run the test assembly. It's same with running code UI test via MSTEST command line on your local machine.
Update
Q: Will the Test Agent 2015 support all the scenarios supported by
Test Controller and Test Agent of Visual Studio 2013?
A: We recommend you use Agents for Visual Studio 2015 in all the new automated testing scenarios. You can use the Deploy Test Agents task in a build definition to download and install the test agents on your machine. The following table shows the scenarios supported by Agents for Visual Studio 2013 and the alternatives for Team Foundation Server (TFS) 2015 and Team Services (TS).
Source Link: Install and configure test agents
Currently, there is no alternative for this scenarion. You may have to use test agent 2013 to achieve it.
In a project using official Microsoft unit test framework I can put test in categories however I can't run these categories from the test explorer in Visual Studio.
I specifically want to set one category to Run after build (not on a build server via CI but locally) because I might have specific test I do not want to run that often.
Is there any way to do this or is the testing aspect from within Visual Studio so limited?
When running tests from the test explorer window, you can filter by "Trait".
Entering Trait:"Important" into the search bar will select only tests marked with [TestCategory("Important")].
To exclude certain tests prefix with '-' e.g. -Trait:"Slow"
You can achieve this using vstest (part of Visual Studio), either via CI (Jenkins, etc.), a Windows Scheduled Task, launched batch file, etc.
Simply call something like:
Vstest.console.exe myTestProject.dll /TestCaseFilter:TestCategory=Nightly
Please reference:
https://msdn.microsoft.com/en-us/library/dd286683.aspx
We use TFS 2012
Through a build definition we execute automated tests from MTM test plans.
How can we ensure TFS uses vstest.console.exe to run each test?
I've read online from other posts that TFS 2012 let's you choose either MSTest or VSTest as the test runner.
I've also read that if you use MTM you effectively are locked into using a test settings file(.testsettings) which means you are locked into using MSTest. This seems to go against TFS 2012 giving you a choice.
Test Manager currently is only supporting MsTest. This is a limitation in Test Manager. It's the same issue that prevents you from associating a xUnit or NUnit test automation on a test case.
So while TFS 2012 gives you the choice, each choice comes with its own set of limitations.
To be clear:
New test runner: Work in Continuous Integration, you specify which tests to run by selecting the new agile test runner (in TFS 2013 this is a fixed choice in the default templates, template customization is required to run mstest). It can run tests during build and you can directly invoke it from the commandline if you want to run tests after deployment. You can use a .runsettings file to specify certain options it should pick up. This option is required to execute 3rd party test frameworks like NUnit, XUnit.NET, Chutzpah etc.
The MTM test runner: Execute automated tests that are associated to a Test Case work item or execute tests using the Test Agent to run tests from a remote system using the Team Test infrastructure. You can use a .testsettings file to specify certain options it should pick up. These tests will be executed using MsTest and cannot contain 3rd party test frameworks.
In your Continuous Integration build you can configure multiple Test Runs, each can be configured to a specific test framework. So you can have both options in your build.
More on the differences and how you can use the different settings files.
Over time all test options will be moved to the new test runner. In TFS 2013 the option to configure a MsTest based test run in your Build Definition has been removed by default. You will need to customize the build process to select a different test runner.
I have created some unit tests in Visual Studio 2012 that I need to test out some API functionality. The API is used to create plugins that run within another application.
Does anyone know how I can run Unit Tests outside of the Visual Studio Environment i.e. not using the Test Explorer. The Unit Tests need to be called by a Test Plugin that is run by the main application and in the same process space.
Hope somebody can help.
I assume you running MSTest.
You can run tests outside VS using MSTest.exe.
Call this exe from your plugin with appropriate command line options.
The NUnit Test Adapter for VS 2012 allows you to run NUnit tests from within the Visual Studio 2012 IDE via the Test Explorer. However, the reason I have NUnit tests in the first place is because I have tests written against SharePoint 2010 (I know, these are Integration Tests, not Unit Tests), which, due to the requirement that the SharePoint API be accessed through a 64-bit client means that MSTest/VS can't run them.
I was hoping that with the NUnit Test Adapter, however, I would be able to run the SharePoint tests from within Test Explorer in Visual Studio but I keep getting the error message:
SetUp failed for test fixture xyz
SetUp : System.IO.FileNotFoundException : The Web application at http://sp could not be found. Verify that you have typed the URL correctly. If the URL should be serving existing content, the system administrator may need to add a new request URL mapping to the intended application.
This indicates to me that the NUnit Test Adapter is running as a 32-bit process (I suspect the process is vstest.executionengine.x86.clr20.exe).
Is there any way to run NUnit tests that target SharePoint 2010 using the NUnit Test Adapter and Test Explorer in Visual Studio 2012?
To run tests in 64-bit process:
In Visual Studio 2012, select TEST->Test Settings->Default Processor Architecture->x64 menu item
With vstest.console.exe, specify /Platform:x64 command line option.
You may also be interested in SharePoint Emulators, which allow you run SharePoint integration tests in isolation.