I am currently working on Microsoft Test Manager. We are not using MTM for any automation, what we do is just pass or fail the test cases manually. Please suggest me if there any possible way for start automating my test cases. If there is any mechanism for record and playing the test cases, that's what i want.
I am in search for getting a guideline to start the automation with the help of MTM. Can anyone help me to start this thing?
This is two (or three) step process
Creating Coded UI test as mentioned above. (Andrew)
a. You have to create Coded UI project using VS ultimate version (I hope that you have it already)
b. Create your test (Eg EnvironmentCheck)
Using automated code using the recorder (Generate code for Coded UI Test) or
Using own code by have [TestMethod] attribute added on top of each test method as follows
[TestMethod]
public void EnvironmentCheck()
{
// your code here
}
More detail can be found on stack overflow question Hand Coding Coded UI Tests c. Verify you can test your method with Visual studio ( Right click inside the method and Click 'Run Tests' or 'Debug Tests')
d. Then you submit your code to TFS Build server and build it successfully
Create Test case at MTM (Microsoft Test Manager)
a. Create Test suite and new Test case at the mtm (as you did on manual test case)
b. New test case name can be 'EnvironmentCheck' and note down the ID (work ID)
c. Associate your work ID with test method using visual studio
More info msdn ==> This will make your test case to automated state at the mtm.
(Create Lab environment or VMs - This is another process. More on msdn)
d. Create a test settings or lab setup
e. Execute the test by clicking Run or Run with options
This will be overall steps, but there will be more steps based on your what you want to automate test. You can refer more on book mentioned by Elena.
What you're looking for are called coded UI tests. You start by creating an action recording of your manual test and then creating a coded UI test from that action recording. Do a google search for "coded UI" and you'll find lots of information.
There is a lot of MSMD articles describing this topic but if you want a guide here is a e-book to download for free that could help you to start:
Testing for Continuous Delivery with Visual Studio 2012
PS: If you are using Visual Studio 2010 this book will be helpful, too, since the approach is similar.
Related
We have an established CI process which links 118 Ranorex test cases to MTM test cases and runs them on a dedicated automation VM, which is triggered by a timed build in Visual Studio.
The CI process is still working correctly, but I have just tried to add another test and found that due to changes in the layout of the test case work item page in VSO/MTM, the 'Associated Automation' function is no longer shown.
So you used to click on the ellipses and the list of test cases to be linked would appear: VS CodedUI MTM_VSO work item linking
As you can see in this image, the 'Associated Automation' tab has moved and the linking mechanism is no longer present: Missing MTM_VSO test case mechanism
I think it is tied into this recent Microsoft VSO update hxxps://www.visualstudio.com/en-us/news/2015-dec-10-vso (sorry for posting the link this way but I can't have more than 2 links) but I wondered if anyone else had encountered this issue and how to resolve it.
Sorry for the long post, but I wanted to make everything clear.
There is already a feedback submitted for this on Visual Studio page, refer to this link for details: https://connect.microsoft.com/VisualStudio/Feedback/Details/2137069
Currently, you can create the test case from your test assemblies via following steps first and then update the test cases.
Run "Developer Command Prompt" and navigate to the debug folder via "cd" command. For example: cd C:\xxx\xxx\solution\testproject\bin\debug
Create test case for the test assemblies via "tcm testcase" command: tcm testcase /collection:https:xxxxx/DefaultCollection /teamproject:yourproject /import /storage:xxx.dll
Is there an elegant way to setup StartUp and Cleanup methods to run for a Load Test in Visual Studio ?
I have a Controller - Agents system for load testing and I need to find a way to run a method before the Test runs and after it finishes but only in the controller.
Plugins can be written for load tests. Basically the plugin provides code to be run during the initialisation of the test run. That code can attach event handlers for the following and hence the test can be manipulated in many ways. (List of events copied from here.)
LoadTestStarting, LoadTestFinished, LoadTestWarmupComplete, TestStarting, TestFinishedTestSelected, ThresholdExceeded, HeartBeat, and LoadTestAborted.
See this Microsoft page (and the pages it links to) for more details on writing load test plugins.
Visual Studio tests also use a ".testsettings" file to specify several aspects of the test. This file allows set-up and clean-up scripts to be specified. These can be seen in the image below "Configure the test controller and roles for remote data collection and execution" and in the "Test Settings: Setup and Cleanup Scripts" parts of this Microsoft page.
Not part of the question, but added for completeness. Plugins can also be written for the web tests. Two forms are provided, web test request plugins and http://msdn.microsoft.com/en-us/library/ms243191(v=vs.110).aspx plugins.
See also this Stackoverflow question VS2010 Load Testing: How can I perform custom action that is run once prior to each load test
I remember running a set of test cases and generating a report (.html) using Selenium and also TestNG. I would like to know how to do this for a set of .webtest files that I recorded using Visual Studio 2013. I can run them individually but how about one by one automatically and getting a final result that says which tests failed.
Thank you.
Create a load test in Visual studio, add all the wanted tests, set a constant user load of 1. Then all the tests will run one after another. The load tests results will include the pass/fail counts. Setting a higher use load may be useful, depending on your application, and the whole test will run faster.
The "Details" page will show all the executed tests on a time-graph, a tick box allows the failed tests to be shown in red.
If you set each web test to stop on the first error then the error page in the results should name precisely the tests that failed, and the reason.
The "Summary" page will give a summary of the results, it may be copied (use right-click context menu) into Word document and then edited as needed.
Which test is executed and when is chosen according to the "Test mix model" by Visual Studio, some of these models are based on random numbers. With a user load of one it may be best to select the "Test mix based on sequential test order". The "Test mix percentage based on the number of tests started" should also work but I can imagine occasionally getting a test run where one test is executed twice and another not executed. If more than one virtual user is used then the "Test mix based on sequential test order" is probably the wrong choice. See here for more on test mix models.
I have an ordered test that contains 4 web tests. The problem is that Visual Studio seems to have a problem loading the test results for an individual test.
I included a screenshot to supplement for the 1000 words. :)
As you may see the 3rd test, called intuitively Webservice03, failed. The screenshot above was taken after I double-clicked on the corresponding line to open the detailed information about that test run. The panels with requests and other useful information are empty.
The issue is quite annoying, because the test is part of a bigger test suite that runs periodically, so when it fails, I have to manually disable the setup and cleanup scripts from the test configuration, run them manually, then run each of the tests manually, instead of just comfortably opening the test results.
Did anyone experienced this problem?
Thanks.
Version details:
Microsoft Visual Studio 2010 Ultimate
Version 10.0.40219.1 SP1Rel
Microsoft .NET Framework
Version 4.0.30319 SP1Rel
I had the same problem and tryed to clarify it via Microsoft forum:
Web test details are not displayed when ran from an ordered test
According to the answer you can not have detailed test result for a web test run as a part of an ordered test.
Here is an alternative to Ordered Tests:
Add the web tests to a load test
In the test mix choose 'Test mix based on sequential test order'. This will achieve the same thing as an Ordered Test.
In the load test run settings, make sure 'Save Log on Test Failure' is set to True:
Set a Constant Load Pattern of 1 user
At the end of the "load" test you will be able to see the number of failures and on the Tables view you can click on the error count which takes you to a dialog that lets you browse the individual web test failures.
Is there a way to integrate Visual Studio 2010 Coded UI Test with Specflow?
I am not sure if these 2 can hook somehow.
I have this idea to Create a Data-Driven Coded UI Tests by using some Coded UI API (I don't know yet what) and hook it up into Specflow.
I have to revisit the video Specflow guys had last year to see how all can fit together
and I would appreciate some input, ideas.
Does this has to do with creating unit test providers for Specflow like the one below (since Coded UI work in VS 2010):
MsTest.2010 MsTest version for .Net 4.0. Supporting test-categories.
Will unitTestProvider (generatorProvider and generatorProvider) be that connection?
Or simply these 2 won't mix and are competing or different ways to do integration or acceptance testing.
Is Coded UI Tests actually close to running Specflow via selenium for UI automation test
with additional benefit that the code is generated for you.
I just want o gain the benefit of Coded UI Tests (maximize its feature both at design and run time) and Specflow (again design and run time)
(More about this on Microsoft site)
Create a Data-Driven Coded UI Test
http://msdn.microsoft.com/en-us/library/ee624082(VS.100).aspx
Using Visual Studio Premium or Visual Studio Ultimate, you can create a coded UI test that tests whether the user interface (UI) for an application functions correctly. The coded UI test performs actions on the user interface controls for an application and verifies that specified controls display the correct values. For more information about how to create a coded UI test, see How to: Create a Coded UI Test.
After you create your coded UI test with specified data, you might want to run your coded UI test multiple times with different sets of data to test different conditions. To do this you can add parameter values from a data source to create a data-driven coded UI test. Each row in the data source is an iteration of the coded UI test. The overall result of the test will be based on the outcome for all the iterations. For example, if one test iteration fails, the overall test result is failure.
For more general information about data-driven tests, see How to: Create a Data-Driven Unit Test. (http://msdn.microsoft.com/en-us/library/ms182527.aspx)
Thanks,
Rad
Basically what you need to do is:
Create a custom generator to decorate your test classes properly
Ensure the Coded UI API is initalized during any SpecFlow hooks you call
I have a solution on how to do both of these on the SpecFlow wiki: https://github.com/techtalk/SpecFlow/wiki/Using-SpecFlow-with-CodedUI-API