I'm trying to set up a build server script to run VS Load Tests. My preferred build server is Team City, but I would accept VSO's build server as well.
Thanks!
You should not use build server to run load test.
Visual Studio Online provides cloud-based load test, you can use it for your testing.
Instruction about VSO Load Test: https://www.visualstudio.com/en-us/get-started/test/load-test-your-app-vs
Related
I have created a continuous integration in dev.azure pipeline. I have created a build>>release>> added Agent as VSTS Test Platform Installer >> added Visual Studio Test runner which will use vstest.console.exe to execute webtests.
I have a solution>project>webtest scripts and supported files which i ran locally successfully in Visual Studio 2017. Now i am trying to move these web tests to pipeline such that i can run them as CI.
I have added everything as below in dev azure pipeline:
Added Visual Studio Test Platform Installer. vsTestVersion: 15.0
Configured assembly test files as ***test*.dll
***.WebTests
***.testsettings.
Search folder as: $(System.DefaultWorkingDirectory)/_Performance/drop/
Version as: Visual Studio 2017
Settings file as: $(System.DefaultWorkingDirectory)/_Performance/drop/Local.testsettings
Build platform: AnyCPU and rest settings as runInParallel: false
codeCoverageEnabled: true
testRunTitle: 'LoadTest_$(rev:r)'
platform: AnyCPU
configuration: Release
continueOnError: true
Now when i am saving my release and running it. It runs the whole solution and copy all files successfully in the virtual drive under D:\a\r1\a_Performance\drop
But as a next step once the test runs after VSTS Installer it shows.......................................
............
No test is available in D:\a\r1\a_Performance\drop\WebAndLoadTestProject1\bin\Release\Microsoft.VisualStudio.QualityTools.WebTestFramework.dll D:\a\r1\a_Performance\drop\WebAndLoadTestProject1\bin\Release\WebAndLoadTestProject1.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
2019-03-04T20:37:35.7477518Z
enter image description here
I figured out a quick solution. Convert the .webtest file to a code test file and checkin that code in git. Run the same solution now in azure DevOps release. It is running the same test successfully in the assigned vsts2017 VM. So as per my understanding it seems Visual Studio webtests cannot run directly in pipeline release.
To convert the webtests into code open the vsts webtest on the toolbar and there is a Generate Code button in bottle shape. Clicking this button prompts for a test name, and then generates code of the same webtest. Both the webtest and code are separate from each other. Changes in one will not added into other.
I've got some CodedUI tests I'd like to run on a remote machine from visual studio. Now it used to be that you'd install a test controller somewhere, a test agent on the box you want to run on, get them talking, then use a test settings file that was pointed to that controller. I was going through this and installed agents for 2015 update 3 on my run box. I noticed that there is no controller configuration option. After some googling I found this: https://vstfsalm.wordpress.com/2015/08/18/tfs-2015-test-agent-has-not-been-configured-run-the-test-agent-configuration-tool-to-configure-the-test-agent/. Very cool. So I go in to tfs to set up a machine group. Great. The part I don't get is: how can I now tell VS to run at that group? Test settings files require a controller when they are set to remote run. Do I have to downgrade the agents back to 2013? Does anyone know if Microsoft just didn't implement this work style in 2015 (on purpose or otherwise) or if it just isn't implemented yet?
To run the code UI test via vNext build on TFS update3. First make sure the code UI test can be executed successfully in your Visual Studio on your remote machine locally. Also don't forget to configure the build agent to interact with desktop.
Then you just need to add Deploy TestAgent on RemoteTestMachine task and run Functional test. When you queue a build, it will deploy your code ui test solution to the drop folder on the build server machine. Then use MSTEST to run the test assembly. It's same with running code UI test via MSTEST command line on your local machine.
Update
Q: Will the Test Agent 2015 support all the scenarios supported by
Test Controller and Test Agent of Visual Studio 2013?
A: We recommend you use Agents for Visual Studio 2015 in all the new automated testing scenarios. You can use the Deploy Test Agents task in a build definition to download and install the test agents on your machine. The following table shows the scenarios supported by Agents for Visual Studio 2013 and the alternatives for Team Foundation Server (TFS) 2015 and Team Services (TS).
Source Link: Install and configure test agents
Currently, there is no alternative for this scenarion. You may have to use test agent 2013 to achieve it.
We use TFS 2012
Through a build definition we execute automated tests from MTM test plans.
How can we ensure TFS uses vstest.console.exe to run each test?
I've read online from other posts that TFS 2012 let's you choose either MSTest or VSTest as the test runner.
I've also read that if you use MTM you effectively are locked into using a test settings file(.testsettings) which means you are locked into using MSTest. This seems to go against TFS 2012 giving you a choice.
Test Manager currently is only supporting MsTest. This is a limitation in Test Manager. It's the same issue that prevents you from associating a xUnit or NUnit test automation on a test case.
So while TFS 2012 gives you the choice, each choice comes with its own set of limitations.
To be clear:
New test runner: Work in Continuous Integration, you specify which tests to run by selecting the new agile test runner (in TFS 2013 this is a fixed choice in the default templates, template customization is required to run mstest). It can run tests during build and you can directly invoke it from the commandline if you want to run tests after deployment. You can use a .runsettings file to specify certain options it should pick up. This option is required to execute 3rd party test frameworks like NUnit, XUnit.NET, Chutzpah etc.
The MTM test runner: Execute automated tests that are associated to a Test Case work item or execute tests using the Test Agent to run tests from a remote system using the Team Test infrastructure. You can use a .testsettings file to specify certain options it should pick up. These tests will be executed using MsTest and cannot contain 3rd party test frameworks.
In your Continuous Integration build you can configure multiple Test Runs, each can be configured to a specific test framework. So you can have both options in your build.
More on the differences and how you can use the different settings files.
Over time all test options will be moved to the new test runner. In TFS 2013 the option to configure a MsTest based test run in your Build Definition has been removed by default. You will need to customize the build process to select a different test runner.
All of the load tests documentation on MSDN describes a computer (running visual studio) connecting to a test controller, which connects to the test agents. Is it possible to run Visual Studio on the Test Controller, and run tests from there?
Running Visual Studio on a Controller with agents is more or less like running local test. You can run load tests from a single (developer?) machine, but you usually can't generate enough traffic to really see how the application responds, especially if the target is hosted on the same machine. It's also very depending on your context.
Using Controllers & Agents in a test rig, provides a distributed architecture to generate a lot of load and activity.
I have a test project that I want to automate on a test server. For now, I have installed the Visual Studio agents (Test Controller, Test Agent and Test Lab) on my local machine, as I hope to try it out here before touching anything on the test server.
The problem I'm having is that I can't find adequate documentation on automating tests using the Visual Studio agent tools. I have successfully set up and configured the test agent and test controller on my local machine, but I'm not sure where I go from here. The test controller is connected to Team Foundation Server. I have a test project contained within TFS - I want to run the tests within it.
I tried launching the VS test manager and attempted to find a way to run tests from there, but I was unsuccessful.
How do I go about doing this? (in simple steps)
My previous experience is with NUnit, and launching automated tests was as simple is launching the command line NUnit program, providing it with the path of the test project and a few other parameters.
Once the Visual Studio agents are set up and configured correctly, it's simply a matter of running tests against a .testsettings file which is connected to the test controller.
When the tests are run (either via the Visual Studio GUI or the mstest.exe command-line program), the active testsettings file will run the tests on any test agent which is connected to the test controller.
So in terms of automating tests, it's just a matter of specifying the correct testsettings file as one of the command-line switches to the MSTest.exe.
A testsetting file can either be created within the Visual Studio solution or alternatively they can be created within Microsoft Test Manager (automated tests can also be scheduled using the MS Test Manager command-line program, tcm.exe).
To select an active testsetting file to use when running tests from Visual Studio 2010, select the 'Test' menu on the toolbar, then 'Select Active Test Settings'.