I am working with Visual studio Load test. I want to prepare Excel report after successful Load test. I want to trigger a Exe or create custom C# class inside the Load test solution for the report generation. But for both i need the test ending event. Is there anyway to find out Report completion?
Thanks in advance,
Subbiah K
A load test plugin can run at the end of the test. Its documentation is not clear on whether that means after the tests have executed or after all the results have been saved. I am not aware of any way of "triggering" the Excel report generation by program.
If you run the load test by using mstest.exe or a similar program then you can follow that by a command by another that you write to do the reporting you require.
The ".testsettings" file has an option for a command (batch or exe etc) to be run after a test completes. I have not tried using it with a load test but it may provide a place to call your reporting program.
Related
I have a solution where I am running various load test scenarios via the command line for Visual Studio 2017. I know I can use MSTest to do this, and am using /testcontainer arg for picking which .loadtest file, and the Set Test.UseRunSetting= WhateverRunSetting arg to pick which run setting I want (controller duration, context parameters, etc).
I can't seem to find a way to change the Step Load Pattern from command line, however. This would allow me to, from the command line, set things like ramp up time, max users, initial users, and other flags. The other way I can think to do this is if I can say which scenario to run (instead of all scenarios in the .loadtest file) based on a command line arg.
Well, the route I used was to write a Powershell script that accepts the parameters I want, such as initial users, step duration/ramp time, context parameters, max users, etc. Then takes those and edits my .loadtest file (since it's an XML file) and saves the .loadtest with the edits, including setting it as the active run-setting.
Then using MSTest as noted above, I can make sure that run setting and .loadtest is used and kick off my test(s).
Another way to do this is use of plug-ins. You can create a plug-in in your load test. The plug-in will read the external excel or xml file to control the user load. You can modify the excel or xml through automation.
I have same thought as yours for modifying load test files externally. Never thried though.
I'm using Visual Studio Online Load Testing to test an API with variable parameters coming from a CSV file.
My setup looks like this:
In properties I set "Show Separate Request Results" to True, hoping that I would be able to see which parameters were used during the test, but I cannot find anything on this in the report?
Is this the way to do this or am I doing something wrong?
Visual Studio load tests are not great at showing how individual test cases worked. The test case logs show the data source values used by a test, look in the context section of the log. By default, logs of the first 200 test cases that fail are retained; altered via Maximum test logs in run settings. Logs of successful tests can also be retained by altering Save log frequency for completed tests in run settings.
Whilst the log files have the data in their context sections, it is hard work (ie lots of mouse waving and mouse clicking) to open each log file, view the context, scroll the right section into view, close the log file, etc.
The mechanism I use to record data source usage etc is to have a web test plugin with a PostWebTest method. It writes useful data to a simple text file as each test case finishes. I write one line per test case, formatted as a CSV so it can easily be read and analysed in a spreadsheet. The data written includes date, time, test outcome, some data source values, and some context parameter values extracted or generated during the run. Tests run with multiple agents will get one file written on each agent. Gathering these files will be a little work but less than viewing individual test case log files. Unfortunately I have not found a way of collecting these files from load tests run with Visual Studio Team Services (previously known as Visual Studio Online).
An early version of the plugins I wrote can be found here.
When create a new test project using visual studio these 2 files are created, what's the purpose of these files? If to run the tests using Visual Studio, these files apparently is not needed.
Local.testsetting mainly used to define your test attributes. Lets assume you have set of test cases that you want to execute using a remote controller machine that you can configure in 'Host' section in this setting file. Also there are multiple other test attributes like 'TestTimeout: to set specific timeout for all your test cases, 'Setup & Cleanup Script': if you wish to run some script before running any test and after all your test cases executes. You can take this file as a Global setting file for your all test cases within this test project.
There will be another file with extension .VSMDI. This file contains list of test cases within this test project. And you can group test cases using this. For example, lets assume you have set of test cases that you want to integrate in your automatic build system and you have some other test cases (Like something opening a browser and check for some element) that you don't want to integrate with your build system. So can do all this grouping using this .vsmdi.
-Thx
I want to solve a problem with a random failures on Visual Studio 2010 Coded UI tests. Is it possible to configure mstest to try each failed test again before failing it in the results? Thank you.
You could make the test(s) in question data-driven. For example, use an XML file with a bunch of blank test rows so that the test runs multiple times. It will report the success/failure of each test-row individually. You don't actually have to make the test(s) extract any data from the XML file.
http://callumhibbert.blogspot.com/2009/07/data-driven-tests-with-mstest.html
I need some advice as to how I easily can separate test runs for unit tests and integration test in Visual Studio. Often, or always, I structure the solution as presented in the above picture: separate projects for unit tests and integration tests. The unit tests is run very frequently while the integration tests naturally is run when the context is correctly aligned.
My goal is to somehow be able configure which tests (or test folders) to run when I use a keyboard shortcut. The tests should preferably be run by a graphical test runner (ReSharpers). So for example
Alt+1 runs the tests in project BLL.Test,
Alt+2 runs the tests in project DAL.Tests,
Alt+3 runs them both (i.e. all the tests in the [Tests] folder, and
Alt+4 runs the tests in folder [Tests.Integration].
TestDriven.net have an option of running just the test in the selected folder or project by right-clicking it and select Run Test(s). Being able to do this, but via a keyboard command and with a graphical test runner would be awesome.
Currently I use VS2008, ReSharper 4 and nUnit. But advice for a setup in the general is of course also appreciated.
I actually found kind of a solution for this on my own by using keyboard command bound to a macro. The macro was recorded from the menu Tools>Macros>Record TemporaryMacro. While recording I selected my [Tests] folder and ran ReSharpers UnitTest.ContextRun. This resulted in the following macro,
Sub TemporaryMacro()
DTE.Windows.Item(Constants.vsWindowKindSolutionExplorer).Activate
DTE.ActiveWindow.Object.GetItem("TestUnitTest\Tests").Select(vsUISelectionType.vsUISelectionTypeSelect)
DTE.ExecuteCommand("ReSharper.UnitTest_ContextRun")
End Sub
which was then bound to it's own keyboard command in Tools>Options>Environment>Keyboard.
However, what would be even more awesome is a more general solution where I can configure exactly which projects/folders/classes to run and when. For example by the means of an xml file. This could then easily be checked in to version control and distributed to everyone who works with the project.
This is a bit of fiddly solution, but you could configure some external tools for each of group of tests you want to run. I'm not sure if you'll be able to launch the ReSharper test runner this way, but you can run the console version of nunit. Once you have of those tools setup, you can assigned keyboard shortcuts to the commands "Tools.ExternalCommand1", "Tools.ExternalCommand2", etc.
This wont really scale very well, and it's awkward to change - but it will give you keyboard shortcuts for running your tests. It does feel like there should be a much simpler way of doing this.
You can use a VS macro to parse the XML file and then call nunit.exe with the /fixture command line argument to specify which classes to run or generate a selection save file and run nunit using that.
I have never used this but maybe it could help....
http://www.codeplex.com/VS2008UnitTestGUI
"Project Description
This project is about running all unit test inside multiple .NET Unit tests assembly coded with Visual Studio 2008."