We are using Microsoft agents & controllers to execute many CodedUI tests in parallel across many Virtual Machines. This produces many .trx results files. Is there anyway to merge all of our TRX files back into one trx file?
Thanks!
Related
I'm currently working within a monolithic solution that is a conglomeration of several smaller projects and the result of many developers efforts. However, one of these efforts was not the creation of a standards document regarding naming practices for test classes. As such I have over 1500 individual test classes that may have a suffix of "Test", "Tester", or "Tests".
Is there a good way to batch rename all the *[Test|Tester].cs files within our solution to *Tests.cs taking into account the preexisting TFS and project file structure?
The solution that seemed to work for me was to do a Find & Replace via a regular expression across targeted test projects within my solution.
In the standared Visual Studio Find and Replace Dialog
I searched for public\W+class\W+([a-zA-Z0-9-_]+)Test(er)?\b
and replaced with the captured group of public class $1Tests
within all files matching *Test.cs;*Tester.cs
Importantly, and for reasons unknown to me, if I didn't check the "Keep modified files open after Replace All" the find and replace would fail
eg:
Once the find and replace operation was completed I used Resharper's Move Types Into Matching Files refactor function on the test packages in order to update references in both TFS and various project files.
Use a local workspace and your bulk renaming tool of choice. It doesn't have to be within Visual Studio. A local workspace will allow changes to the structure of the project to be made from any tool and still "noticed" by the pending changes window or tf.exe.
You'll still have to update the *.*proj files to take into account that the files have been renamed, of course.
I have created a number of (separate) CodedUI projects within Visual Studio 2013, in order to test the basic functions of a website.
Test-cases are created as separate projects, as I expect some (or all?) of them to change over time, and thus to ensure 'modularity' of capture for ease of subsequent maintenance.
Now, I see I can easily create an Ordered Test within each project, which will allow the same test-case to be run and re-run as many times as I wish; But, it's not obvious to me how I can create an Ordered Test whereby I can add different test-cases created as different projects. Certainly, not directly.
Is it possible?
Also, can I rename the Ordered Test list and save it to a separate folder where I can store differing Ordered Tests to test functionality, as I wish?
Ideally, I'd like to create an Ordered Test external to any specific project, so I can go into any project I wish and add whatever tests I wish, as the test-environment is always the same.
you should have created a single project for your application. To ensure 'modularity', coded ui has given us the option of creating different UI Maps within a same project. Each UI MAP will represent a module of your application. This approach will give you easy maintenance and it will also help you to create ordered test cases which contain test cases from different UI Maps.
for more details please see this link
https://msdn.microsoft.com/en-us/library/ff398056.aspx
Thanks
Yes, I sort of see that. And I guess it's easy enough to move the code to become separate 'solutions' within a 'project'.
However, I want to work with TFS server too, so will look at the MTM route as well.
But it may be that I need my captured CodedUI to be 'solutions' within a single project too - though I really want my modules to be 'stand-alone' projects for safe-keeping.
Will investigate further.
I have a visual studio solution of C# projects. Currently, I do not have test projects in my solution. So I do not have file *.vsmdi.
I want to test my projects via command line like this.
MSTest.exe /testmetadata:*.vsmdi /resultsfile:Logfile.log
Is it possible to create .vsmdi without adding project? if possible, how?
To execute in command line, *.vsmdi file MUST be present in the solution?
If I have to test using IDE, the I should have *.vsmdi file in test project?
The only use of *.vsmdi file is for listing the test cases?
1) Technically it’s possible to create a .vsmdi file without adding your test project(s) to your solution. Since the content of the VSMDI files is XML you can write them by hand or generate them with a tool based on the tests in your assembly. However, VSMDI files are less useful when the test projects are not included in the solution because the power of the VSMDI files is managing your tests. But you can still use them (see also answer to question 2).
If you want to write the vsmdi file by hand you need to hash the FQN with SHA1 and create a Guid of the first 128 bits of the hash as shown below (code from Dominic Hopton). See for more information his blog post: Helpful Internals of TRX (and VSMDI files).
private static HashAlgorithm s_provider = new SHA1CryptoServiceProvider();
internal static HashAlgorithm Provider
{
get { return s_provider; }
}
/// Calculates a hash of the string and copies the first 128 bits of the hash
/// to a new Guid.
///
internal static Guid GuidFromString(string data)
{
byte[] hash = Provider.ComputeHash(
System.Text.Encoding.Unicode.GetBytes(data));
byte[] toGuid = new byte[16];
Array.Copy(hash, toGuid, 16);
return new Guid(toGuid);
}
2) No, the vsmdi file doesn’t need to be in your solution folder (or in your solution) as long as the storage attribute of the TestLink element points to the correct assembly containing the test (relative to your vsmdi file). For example:
<TestLink id="0c257cc9-ffed-cd5e-bda1-56df14ad68c2" name="AddOneAndTwo_Test"
storage="TestProject\bin\debug\testproject.dll" />
3) Vsmdi files exists on the solution level and not at project level. If you have more than one test projects you can use one vsmdi file to manage all your tests. Please note that the storage attribute is ignored when you run the tests from Visual Studio assuming that you loaded the projects in your solution
4) Vsmdi files are used for managing the tests in your solution and you can create a hierarchy of several layers within the VSMDI file. For example you have more than one assembly in your solution you want to test. Under the root test list (called: List Of Tests) you create a test list for each of your projects. You can then divide the test list for one of the projects into two or more separate test lists (e.g. one for testing the business logic and one for testing the error handling). When you only want to test the business logic tests from project A you can select the corresponding test list and run the tests in it.
An extra note about vsmdi files:
A) Since Visual Studio 2012 Microsoft decided to make VSMDI files (and with them the test lists) deprecated. Test lists got replaced by Test Categories (already exists in Visual Studio 2010) and Playlists.
Test categories are attributes you can set on individual test methods and can be used in builds. Playlists were introduced in Visual Studio 2012 Update 2 and are currently not supported in MSBuild.
For more information about Test Categories see: http://msdn.microsoft.com/en-us/library/dd286595.aspx
What's the purose of .vsmdi file? Do I need to check into the source control system?
The VSMDI file is created by Visual Studio when you create a test project for the first time. It contains a list of all tests that Visual Studio can find in your solution assemblies and allows you to divide your tests into so-called test lists. These test lists can be used to categorize your tests and let you select a subset of tests for execution.
You can use this mechanism for running sub-selections. However, you can also (freely) assign multiple test categories to a test, which enables you to achieve the same, in a more flexible way. And with the known issues with VSMDI files, like uncontrolled duplication of these files and obsolete tests being listed with a warning icon, it might seem the better way to do things like this.
My overall suggestion is: check-in your default generated .vsmdi file. This will prevent Visual Studio from (re-)generating such files on your own and your team members systems when new test projects are added. Decide on usage of test lists or assigning categories to tests directly based on your usage experience. Test lists are easy to start with, but less suitable is you want to have flexibility for a large set of tests.
It's used for Testing in Visual Studio. If you don't do testing in Visual Studio, I wouldn't worry about it. But if you do, and you have hundreds of tests it might be worth keeping.
I have a few tests that need to be fed with external data from excel files. The files are included in the test project, and in Visual Studio, I have edited the test settings file (Local.testsettings) to deploy the data files. This makes it work fine i VS.
We are, however, also running continous integration with TeamCity, and in TeamCity this doesn't work. My data files are unavailable to the test. Seems that the tests are run from a temporary folder named "C:\TeamCity\buildAgent\temp\buildTmp\ciuser_AS40VS6 2009-12-11 09_40_17\Out", and the data files are not copied there.
I have tried changing the build action for the data files to "Resource" and setting copy to output dir to "Always", but that didn't help.
Does anyone know how to make this work?
I am running Visual Studio 2010 beta 2 and TeamCity 4.5.5, which is why I'm running MSTest in the first place, and not NUnit...
I get round this by adding my data files (in my case usually XML) as embedded resources and I extract them from the test assembly.
[TestInitialize]
public void InitializeTests()
{
var asm = Assembly.GetExecutingAssembly();
this.doc = new XmlDocument();
this.doc.Load(asm.GetManifestResourceStream("TestAssembly.File.xml"));
}
This post answers this question: MSTest copy file to test run folder
The accepted answer is technically correct. However, from my experience, I find that the embedding files as resources requires an additional step of remembering to set the property "Embedded Resource". This becomes a challenge when you have a large number of data files. Also, with increasing number of data files, the size of the unit test assembly keeps growing . In my case, I had over 500MB of test data files and packing all them into the assembly was not a good idea.
What is the alternative?
Let the data files remain as they are. Do not use DeploymentItemAttribute, do not use embedded resources. Please refer my proposed solution How do I make a data file available to unit tests?