I want to solve a problem with a random failures on Visual Studio 2010 Coded UI tests. Is it possible to configure mstest to try each failed test again before failing it in the results? Thank you.
You could make the test(s) in question data-driven. For example, use an XML file with a bunch of blank test rows so that the test runs multiple times. It will report the success/failure of each test-row individually. You don't actually have to make the test(s) extract any data from the XML file.
http://callumhibbert.blogspot.com/2009/07/data-driven-tests-with-mstest.html
Related
I'm using Visual Studio Online Load Testing to test an API with variable parameters coming from a CSV file.
My setup looks like this:
In properties I set "Show Separate Request Results" to True, hoping that I would be able to see which parameters were used during the test, but I cannot find anything on this in the report?
Is this the way to do this or am I doing something wrong?
Visual Studio load tests are not great at showing how individual test cases worked. The test case logs show the data source values used by a test, look in the context section of the log. By default, logs of the first 200 test cases that fail are retained; altered via Maximum test logs in run settings. Logs of successful tests can also be retained by altering Save log frequency for completed tests in run settings.
Whilst the log files have the data in their context sections, it is hard work (ie lots of mouse waving and mouse clicking) to open each log file, view the context, scroll the right section into view, close the log file, etc.
The mechanism I use to record data source usage etc is to have a web test plugin with a PostWebTest method. It writes useful data to a simple text file as each test case finishes. I write one line per test case, formatted as a CSV so it can easily be read and analysed in a spreadsheet. The data written includes date, time, test outcome, some data source values, and some context parameter values extracted or generated during the run. Tests run with multiple agents will get one file written on each agent. Gathering these files will be a little work but less than viewing individual test case log files. Unfortunately I have not found a way of collecting these files from load tests run with Visual Studio Team Services (previously known as Visual Studio Online).
An early version of the plugins I wrote can be found here.
I am working with Visual studio Load test. I want to prepare Excel report after successful Load test. I want to trigger a Exe or create custom C# class inside the Load test solution for the report generation. But for both i need the test ending event. Is there anyway to find out Report completion?
Thanks in advance,
Subbiah K
A load test plugin can run at the end of the test. Its documentation is not clear on whether that means after the tests have executed or after all the results have been saved. I am not aware of any way of "triggering" the Excel report generation by program.
If you run the load test by using mstest.exe or a similar program then you can follow that by a command by another that you write to do the reporting you require.
The ".testsettings" file has an option for a command (batch or exe etc) to be run after a test completes. I have not tried using it with a load test but it may provide a place to call your reporting program.
When create a new test project using visual studio these 2 files are created, what's the purpose of these files? If to run the tests using Visual Studio, these files apparently is not needed.
Local.testsetting mainly used to define your test attributes. Lets assume you have set of test cases that you want to execute using a remote controller machine that you can configure in 'Host' section in this setting file. Also there are multiple other test attributes like 'TestTimeout: to set specific timeout for all your test cases, 'Setup & Cleanup Script': if you wish to run some script before running any test and after all your test cases executes. You can take this file as a Global setting file for your all test cases within this test project.
There will be another file with extension .VSMDI. This file contains list of test cases within this test project. And you can group test cases using this. For example, lets assume you have set of test cases that you want to integrate in your automatic build system and you have some other test cases (Like something opening a browser and check for some element) that you don't want to integrate with your build system. So can do all this grouping using this .vsmdi.
-Thx
What's the purose of .vsmdi file? Do I need to check into the source control system?
The VSMDI file is created by Visual Studio when you create a test project for the first time. It contains a list of all tests that Visual Studio can find in your solution assemblies and allows you to divide your tests into so-called test lists. These test lists can be used to categorize your tests and let you select a subset of tests for execution.
You can use this mechanism for running sub-selections. However, you can also (freely) assign multiple test categories to a test, which enables you to achieve the same, in a more flexible way. And with the known issues with VSMDI files, like uncontrolled duplication of these files and obsolete tests being listed with a warning icon, it might seem the better way to do things like this.
My overall suggestion is: check-in your default generated .vsmdi file. This will prevent Visual Studio from (re-)generating such files on your own and your team members systems when new test projects are added. Decide on usage of test lists or assigning categories to tests directly based on your usage experience. Test lists are easy to start with, but less suitable is you want to have flexibility for a large set of tests.
It's used for Testing in Visual Studio. If you don't do testing in Visual Studio, I wouldn't worry about it. But if you do, and you have hundreds of tests it might be worth keeping.
I have data files used as input to my unit tests. These files are quite big and I don't want to copy them each time unit tests are executed. Tests are executed without deployment. So I can just put them into folder under my solution, and... how to obtain path to my solution (or test project source code) when unit test is executing?
Because you can run a test project in different ways (TD.NET, Visual Studio, R# etc.), the path used to reference the tests can change.
For this reason, I embed test needed files in my test assembly and draw them out from there.
You can use:
Assembly.GetExecutingAssembly().Location
in your tests to get the path of the assembly containing the unit tests.
Simple, make the location of the files configurable (and testable).
Then either, set it in the unit testing code or set it thru some config file.