I have an ordered test that contains 4 web tests. The problem is that Visual Studio seems to have a problem loading the test results for an individual test.
I included a screenshot to supplement for the 1000 words. :)
As you may see the 3rd test, called intuitively Webservice03, failed. The screenshot above was taken after I double-clicked on the corresponding line to open the detailed information about that test run. The panels with requests and other useful information are empty.
The issue is quite annoying, because the test is part of a bigger test suite that runs periodically, so when it fails, I have to manually disable the setup and cleanup scripts from the test configuration, run them manually, then run each of the tests manually, instead of just comfortably opening the test results.
Did anyone experienced this problem?
Thanks.
Version details:
Microsoft Visual Studio 2010 Ultimate
Version 10.0.40219.1 SP1Rel
Microsoft .NET Framework
Version 4.0.30319 SP1Rel
I had the same problem and tryed to clarify it via Microsoft forum:
Web test details are not displayed when ran from an ordered test
According to the answer you can not have detailed test result for a web test run as a part of an ordered test.
Here is an alternative to Ordered Tests:
Add the web tests to a load test
In the test mix choose 'Test mix based on sequential test order'. This will achieve the same thing as an Ordered Test.
In the load test run settings, make sure 'Save Log on Test Failure' is set to True:
Set a Constant Load Pattern of 1 user
At the end of the "load" test you will be able to see the number of failures and on the Tables view you can click on the error count which takes you to a dialog that lets you browse the individual web test failures.
Related
We have an established CI process which links 118 Ranorex test cases to MTM test cases and runs them on a dedicated automation VM, which is triggered by a timed build in Visual Studio.
The CI process is still working correctly, but I have just tried to add another test and found that due to changes in the layout of the test case work item page in VSO/MTM, the 'Associated Automation' function is no longer shown.
So you used to click on the ellipses and the list of test cases to be linked would appear: VS CodedUI MTM_VSO work item linking
As you can see in this image, the 'Associated Automation' tab has moved and the linking mechanism is no longer present: Missing MTM_VSO test case mechanism
I think it is tied into this recent Microsoft VSO update hxxps://www.visualstudio.com/en-us/news/2015-dec-10-vso (sorry for posting the link this way but I can't have more than 2 links) but I wondered if anyone else had encountered this issue and how to resolve it.
Sorry for the long post, but I wanted to make everything clear.
There is already a feedback submitted for this on Visual Studio page, refer to this link for details: https://connect.microsoft.com/VisualStudio/Feedback/Details/2137069
Currently, you can create the test case from your test assemblies via following steps first and then update the test cases.
Run "Developer Command Prompt" and navigate to the debug folder via "cd" command. For example: cd C:\xxx\xxx\solution\testproject\bin\debug
Create test case for the test assemblies via "tcm testcase" command: tcm testcase /collection:https:xxxxx/DefaultCollection /teamproject:yourproject /import /storage:xxx.dll
All -- I am using Visual studio 2013 ultimate edition to run my load tests. I have more than one scenario configured on my load test. Each of the scenario run a different set of unit tests. Is is there a way to selectively run the scenarios ? In other words if there are more than one scenario configured, can I just run one of them and not run the rest ?
If the answer is 'NO', any ideas/suggestions would be very helpful.
Thanks in advance.
Two ways I have found:
One way is to set the user load to a constant zero in the unwanted scenarios.
Another way is to set the "Delay start time" of the unwanted scenarios to longer that the total test duration. This option only makes sense when "User test iterations" is false.
I remember running a set of test cases and generating a report (.html) using Selenium and also TestNG. I would like to know how to do this for a set of .webtest files that I recorded using Visual Studio 2013. I can run them individually but how about one by one automatically and getting a final result that says which tests failed.
Thank you.
Create a load test in Visual studio, add all the wanted tests, set a constant user load of 1. Then all the tests will run one after another. The load tests results will include the pass/fail counts. Setting a higher use load may be useful, depending on your application, and the whole test will run faster.
The "Details" page will show all the executed tests on a time-graph, a tick box allows the failed tests to be shown in red.
If you set each web test to stop on the first error then the error page in the results should name precisely the tests that failed, and the reason.
The "Summary" page will give a summary of the results, it may be copied (use right-click context menu) into Word document and then edited as needed.
Which test is executed and when is chosen according to the "Test mix model" by Visual Studio, some of these models are based on random numbers. With a user load of one it may be best to select the "Test mix based on sequential test order". The "Test mix percentage based on the number of tests started" should also work but I can imagine occasionally getting a test run where one test is executed twice and another not executed. If more than one virtual user is used then the "Test mix based on sequential test order" is probably the wrong choice. See here for more on test mix models.
I am currently working on Microsoft Test Manager. We are not using MTM for any automation, what we do is just pass or fail the test cases manually. Please suggest me if there any possible way for start automating my test cases. If there is any mechanism for record and playing the test cases, that's what i want.
I am in search for getting a guideline to start the automation with the help of MTM. Can anyone help me to start this thing?
This is two (or three) step process
Creating Coded UI test as mentioned above. (Andrew)
a. You have to create Coded UI project using VS ultimate version (I hope that you have it already)
b. Create your test (Eg EnvironmentCheck)
Using automated code using the recorder (Generate code for Coded UI Test) or
Using own code by have [TestMethod] attribute added on top of each test method as follows
[TestMethod]
public void EnvironmentCheck()
{
// your code here
}
More detail can be found on stack overflow question Hand Coding Coded UI Tests c. Verify you can test your method with Visual studio ( Right click inside the method and Click 'Run Tests' or 'Debug Tests')
d. Then you submit your code to TFS Build server and build it successfully
Create Test case at MTM (Microsoft Test Manager)
a. Create Test suite and new Test case at the mtm (as you did on manual test case)
b. New test case name can be 'EnvironmentCheck' and note down the ID (work ID)
c. Associate your work ID with test method using visual studio
More info msdn ==> This will make your test case to automated state at the mtm.
(Create Lab environment or VMs - This is another process. More on msdn)
d. Create a test settings or lab setup
e. Execute the test by clicking Run or Run with options
This will be overall steps, but there will be more steps based on your what you want to automate test. You can refer more on book mentioned by Elena.
What you're looking for are called coded UI tests. You start by creating an action recording of your manual test and then creating a coded UI test from that action recording. Do a google search for "coded UI" and you'll find lots of information.
There is a lot of MSMD articles describing this topic but if you want a guide here is a e-book to download for free that could help you to start:
Testing for Continuous Delivery with Visual Studio 2012
PS: If you are using Visual Studio 2010 this book will be helpful, too, since the approach is similar.
is there a way? do I have to wait for building every time I start the test? I want to build from visual studio not from test
thanks
Any time your code changes and you run your test it is going to do a build... so technically you can run your test over and over again and they will only build the first time, but once you run your test why would you run them again without making a code change?
Couple of things that I use that make your test run faster are:
Check the box for "Only build startup projects and dependencies on Run", located Options->Projects and Solution->Build and Run.
Learn the short cut keys
a. "Ctrl+R, T" Runs test in current context, so if your cursor is inside a test method it will only run that test, but when you do it inside of a non test class it will run all of your test.
b. "Crtl+R, Ctrl+T" Debug test same except debug.
c. Others can be found here, those are 2008 if you need to reference others you can find them via google.
Make sure your test are not calling the database or other time intensive resources, use mocking and stubbing.
Run only small sets of test, ie if I am working in a service class I run only the service class test.
Edit: Reading your question again if you want to build and not from a test you can just go to the menu and click Build->Build Solution or press F6. Also it would be helpful if you indicated which version of visual studio you are using because 2010 is different in the sense that you have to click refresh. Either way are you able to clarify?
This is an old question, but I keep seeing people ask it and the issue is still true in VS2017, and it's also true of other test frameworks (Xunit, etc) run from within VS.
I don't know how to make VS stop building all the time. But I do know how to circumvent the compile - run your tests from a console runner, not from within VS. If you're using ReSharper, it has one.
If you aren't using ReSharper, for MSTest, you can start here. https://msdn.microsoft.com/en-us/library/ms182489.aspx
If you aren't using ReSharper, for XUnit, you can start here. https://xunit.github.io/docs/getting-started-desktop.html#add-xunit-runner-ref
Any changes to source code cause compilation, because in order to run tests VS needs up to date DLL with tests.
If you have already compiled project then you can run test multiple times without compilation.
PS: I run MSTest using TestDriven.NET as for me it is faster.