Attempting to automate a test run via the command-line Microsoft Test Manager utility, tcm.exe.
I'm using the following command:
tcm run /create /title:"Automated UI Tests" /planid:27 /suiteid:721
/configid:10 /settingsname:"UI Test Settings 3"
/testenvironment:"SanityEnvironmentSlemish"
/collection:[CollectionUrl]
/teamproject:Main
(I haven't included the collection URL above).
However, I've been getting the following error: "A test run must be created with at least one test case".
This is a bit of a mystery to me as my test suite already has a test case associated with it, which is associated with an ordered test list.
The reason it wasn't working was not because the test run didn't have a test case associated with it, but because the test case in question had a state of "Error" (due to previous test run failures), instead of a state of "Active".
To resolve this, you can choose either to edit the state via the Microsoft Test Manager GUI by going to the 'Test' tab, right-click on the test case of the particular test suite and click "Reset test to active".
Alternatively, if you are attempting to run tests using the Microsoft Test Manager command-line interface (tcm.exe), you can simply use the '/include' command-line switch - which includes all test cases in a test run, regardless of state.
Related
When I run my XCTests, I'd like to automatically rerun, once, any integration (unit/ui) test that fails. Is this possible?
This would be done in the same test, without having to hit 'run' again on my tests, or any part of my tests. I'm running all my tests from the command line.
I am doing UI tests that use the network to make calls to the server. If there is a significant server problem, the test will/should fail and report an error. However, if it is only a temporary problem with the request, it would be nice to rerun the test automatically and see if it passes. Also, with the current state of Xcode UI testing there are some occasional problems where it will crash for an obscure reason, and it would be nice to rerun the test automatically to see if it passes the second time.
It would be especially nice if it could output what happened, i.e. "The test failed the first time, because of failure getting refreshed snapshot, but passed the second time"
You can now do this in Xcode without any scripts.
Press ⌘ + 6 or select the test assistant view.
Filter for failed tests
Select tests, right click and choose Run x test methods > Run in all classes
(optional) If you want to run the same tests again press ⌃ + ⌥ + ⌘ + G for a quick way to run them right away.
Task list to accomplish this without fastlane and with granular test reruns
create a script to rerun the test and capture the list of failed tests, then rerun the tests using the -only-testing:TEST-IDENTIFIER flag of xcodebuild.
parse the resultBundle or .xcresult to find the failed tests
write code to erase the device or simulator executing the tests
reboot the device if it is a real one to clear any possible dialogs
use XCAttachment to collect things you care about, like app logs
use xchtmlreport to generate a nice webpage from the results bundle with your snapshots and attachments.
Fastlane can make it possible here is the awesome blog post regarding same
Stabilizing the CI By Re-runing Flaky iOS XCUI Tests
it does sound strange, but sometimes it happens that the status of a test case that was run a couple of days ago is automatically reset to active.
Do you have any idea why, or where to start investigating?
I mention that:
we did press save and close after the run (the test case was shown as passed for a couple of days);
the test was not included in any other test suite;
the test is a manual one;
it did not happen for test belonging to suits that were marked as completed;
We are using MTM 2012.
Thanks!
I only know a similar behavior regarding automated tests.
You will observe this if
you associate your automated test run with a CI or nightly build AND
if you set up a *Retention Policy" for these build AND
if "Build Delete Options" are set like this:
In this case the test runs will be deleted along with the build.
But it only applies to automated tests to my knowledge...
The behaviour you are observing is kind of similar one: your most recent manual test run gets deleted for some reason; that is the only reason I can imagine for a test case being reset to active "automatically".
Try to investigate in this direction; means what in your environment can cause manual test run to be deleted?
I remember running a set of test cases and generating a report (.html) using Selenium and also TestNG. I would like to know how to do this for a set of .webtest files that I recorded using Visual Studio 2013. I can run them individually but how about one by one automatically and getting a final result that says which tests failed.
Thank you.
Create a load test in Visual studio, add all the wanted tests, set a constant user load of 1. Then all the tests will run one after another. The load tests results will include the pass/fail counts. Setting a higher use load may be useful, depending on your application, and the whole test will run faster.
The "Details" page will show all the executed tests on a time-graph, a tick box allows the failed tests to be shown in red.
If you set each web test to stop on the first error then the error page in the results should name precisely the tests that failed, and the reason.
The "Summary" page will give a summary of the results, it may be copied (use right-click context menu) into Word document and then edited as needed.
Which test is executed and when is chosen according to the "Test mix model" by Visual Studio, some of these models are based on random numbers. With a user load of one it may be best to select the "Test mix based on sequential test order". The "Test mix percentage based on the number of tests started" should also work but I can imagine occasionally getting a test run where one test is executed twice and another not executed. If more than one virtual user is used then the "Test mix based on sequential test order" is probably the wrong choice. See here for more on test mix models.
I am currently working on Microsoft Test Manager. We are not using MTM for any automation, what we do is just pass or fail the test cases manually. Please suggest me if there any possible way for start automating my test cases. If there is any mechanism for record and playing the test cases, that's what i want.
I am in search for getting a guideline to start the automation with the help of MTM. Can anyone help me to start this thing?
This is two (or three) step process
Creating Coded UI test as mentioned above. (Andrew)
a. You have to create Coded UI project using VS ultimate version (I hope that you have it already)
b. Create your test (Eg EnvironmentCheck)
Using automated code using the recorder (Generate code for Coded UI Test) or
Using own code by have [TestMethod] attribute added on top of each test method as follows
[TestMethod]
public void EnvironmentCheck()
{
// your code here
}
More detail can be found on stack overflow question Hand Coding Coded UI Tests c. Verify you can test your method with Visual studio ( Right click inside the method and Click 'Run Tests' or 'Debug Tests')
d. Then you submit your code to TFS Build server and build it successfully
Create Test case at MTM (Microsoft Test Manager)
a. Create Test suite and new Test case at the mtm (as you did on manual test case)
b. New test case name can be 'EnvironmentCheck' and note down the ID (work ID)
c. Associate your work ID with test method using visual studio
More info msdn ==> This will make your test case to automated state at the mtm.
(Create Lab environment or VMs - This is another process. More on msdn)
d. Create a test settings or lab setup
e. Execute the test by clicking Run or Run with options
This will be overall steps, but there will be more steps based on your what you want to automate test. You can refer more on book mentioned by Elena.
What you're looking for are called coded UI tests. You start by creating an action recording of your manual test and then creating a coded UI test from that action recording. Do a google search for "coded UI" and you'll find lots of information.
There is a lot of MSMD articles describing this topic but if you want a guide here is a e-book to download for free that could help you to start:
Testing for Continuous Delivery with Visual Studio 2012
PS: If you are using Visual Studio 2010 this book will be helpful, too, since the approach is similar.
I have an ordered test that contains 4 web tests. The problem is that Visual Studio seems to have a problem loading the test results for an individual test.
I included a screenshot to supplement for the 1000 words. :)
As you may see the 3rd test, called intuitively Webservice03, failed. The screenshot above was taken after I double-clicked on the corresponding line to open the detailed information about that test run. The panels with requests and other useful information are empty.
The issue is quite annoying, because the test is part of a bigger test suite that runs periodically, so when it fails, I have to manually disable the setup and cleanup scripts from the test configuration, run them manually, then run each of the tests manually, instead of just comfortably opening the test results.
Did anyone experienced this problem?
Thanks.
Version details:
Microsoft Visual Studio 2010 Ultimate
Version 10.0.40219.1 SP1Rel
Microsoft .NET Framework
Version 4.0.30319 SP1Rel
I had the same problem and tryed to clarify it via Microsoft forum:
Web test details are not displayed when ran from an ordered test
According to the answer you can not have detailed test result for a web test run as a part of an ordered test.
Here is an alternative to Ordered Tests:
Add the web tests to a load test
In the test mix choose 'Test mix based on sequential test order'. This will achieve the same thing as an Ordered Test.
In the load test run settings, make sure 'Save Log on Test Failure' is set to True:
Set a Constant Load Pattern of 1 user
At the end of the "load" test you will be able to see the number of failures and on the Tables view you can click on the error count which takes you to a dialog that lets you browse the individual web test failures.