mstest.exe execution via command line with *.vsmdi - visual-studio-2010

I have a visual studio solution of C# projects. Currently, I do not have test projects in my solution. So I do not have file *.vsmdi.
I want to test my projects via command line like this.
MSTest.exe /testmetadata:*.vsmdi /resultsfile:Logfile.log
Is it possible to create .vsmdi without adding project? if possible, how?
To execute in command line, *.vsmdi file MUST be present in the solution?
If I have to test using IDE, the I should have *.vsmdi file in test project?
The only use of *.vsmdi file is for listing the test cases?

1) Technically it’s possible to create a .vsmdi file without adding your test project(s) to your solution. Since the content of the VSMDI files is XML you can write them by hand or generate them with a tool based on the tests in your assembly. However, VSMDI files are less useful when the test projects are not included in the solution because the power of the VSMDI files is managing your tests. But you can still use them (see also answer to question 2).
If you want to write the vsmdi file by hand you need to hash the FQN with SHA1 and create a Guid of the first 128 bits of the hash as shown below (code from Dominic Hopton). See for more information his blog post: Helpful Internals of TRX (and VSMDI files).
private static HashAlgorithm s_provider = new SHA1CryptoServiceProvider();
internal static HashAlgorithm Provider
{
get { return s_provider; }
}
/// Calculates a hash of the string and copies the first 128 bits of the hash
/// to a new Guid.
///
internal static Guid GuidFromString(string data)
{
byte[] hash = Provider.ComputeHash(
System.Text.Encoding.Unicode.GetBytes(data));
byte[] toGuid = new byte[16];
Array.Copy(hash, toGuid, 16);
return new Guid(toGuid);
}
2) No, the vsmdi file doesn’t need to be in your solution folder (or in your solution) as long as the storage attribute of the TestLink element points to the correct assembly containing the test (relative to your vsmdi file). For example:
<TestLink id="0c257cc9-ffed-cd5e-bda1-56df14ad68c2" name="AddOneAndTwo_Test"
storage="TestProject\bin\debug\testproject.dll" />
3) Vsmdi files exists on the solution level and not at project level. If you have more than one test projects you can use one vsmdi file to manage all your tests. Please note that the storage attribute is ignored when you run the tests from Visual Studio assuming that you loaded the projects in your solution
4) Vsmdi files are used for managing the tests in your solution and you can create a hierarchy of several layers within the VSMDI file. For example you have more than one assembly in your solution you want to test. Under the root test list (called: List Of Tests) you create a test list for each of your projects. You can then divide the test list for one of the projects into two or more separate test lists (e.g. one for testing the business logic and one for testing the error handling). When you only want to test the business logic tests from project A you can select the corresponding test list and run the tests in it.
An extra note about vsmdi files:
A) Since Visual Studio 2012 Microsoft decided to make VSMDI files (and with them the test lists) deprecated. Test lists got replaced by Test Categories (already exists in Visual Studio 2010) and Playlists.
Test categories are attributes you can set on individual test methods and can be used in builds. Playlists were introduced in Visual Studio 2012 Update 2 and are currently not supported in MSBuild.
For more information about Test Categories see: http://msdn.microsoft.com/en-us/library/dd286595.aspx

Related

Using project files in NUnit tests

My program deals with input and output files and therefore need to have an access to a folder (included in Visual Studio Project) with existing files, and also create new files and check what's inside.
Tests run on several machines, so absolute path is not an option. Also, I can not self-generate input files.
How I can tell in an NUnit test that I need a folder which is located inside project source tree? NUnit seems to place exe code in some obscure temp folder.
Edit: NUnit 3 is used
Presuming this question refers to NUnit 3, sounds like you need to use TestContext.CurrentContext.TestDirectory.
This gets the directory the assembly was built to, rather that the temp location where it is ran. (Which Environment.CurrentDirectory would return.) Documented here. I believe CurrentContext can also sometimes be null, we use TestContext.CurrentContext?.TestDirectory.
For NUnit 2.X - I believe tests run where they are build, so Environement.CurrentDirectory should be sufficient.

How to bulk refactor class / cs file names in a large TFS managed Visual Studio project

I'm currently working within a monolithic solution that is a conglomeration of several smaller projects and the result of many developers efforts. However, one of these efforts was not the creation of a standards document regarding naming practices for test classes. As such I have over 1500 individual test classes that may have a suffix of "Test", "Tester", or "Tests".
Is there a good way to batch rename all the *[Test|Tester].cs files within our solution to *Tests.cs taking into account the preexisting TFS and project file structure?
The solution that seemed to work for me was to do a Find & Replace via a regular expression across targeted test projects within my solution.
In the standared Visual Studio Find and Replace Dialog
I searched for public\W+class\W+([a-zA-Z0-9-_]+)Test(er)?\b
and replaced with the captured group of public class $1Tests
within all files matching *Test.cs;*Tester.cs
Importantly, and for reasons unknown to me, if I didn't check the "Keep modified files open after Replace All" the find and replace would fail
eg:
Once the find and replace operation was completed I used Resharper's Move Types Into Matching Files refactor function on the test packages in order to update references in both TFS and various project files.
Use a local workspace and your bulk renaming tool of choice. It doesn't have to be within Visual Studio. A local workspace will allow changes to the structure of the project to be made from any tool and still "noticed" by the pending changes window or tf.exe.
You'll still have to update the *.*proj files to take into account that the files have been renamed, of course.

How to refer to test files from Xunit tests in Visual Studio?

We’re using Xunit for testing. We’re running our tests via the built-in Visual Studio 2013 Test Runner, using the Xunit plugin.
The issue is that some of the tests need to refer to files on the filesystem. It seems that Xunit (or the VS Test Runner—not sure which), copies the assembles, but not any supporting files in the bin directory, to another directory before executing the tests, hence our test files are not found. [The MS Testing framework specifies attributes for listing files to be copied, but Xunit does not.]
How to either disable this copying behaviour, or else programmatically determine the original bin/ directory location to fetch the files?
It seems that most proposed solutions (including on the Xunit bug-tracker site) suggest storing the files as embedded resources instead of ‘copy always’ files. However, this is not always practical, for example: testing file manipulation code, and (in one case) code which wants a Sqlite database file.
Okay, typical, just as soon as I post the question, I find the answer myself…
The gist is that the copying (Shadow Copying) of assemblies seems to be done by the .NET framework, not by Visual Studio or by Xunit.
We had been using Assembly.Location to locate the assembly file, and hence the test files. However, this was wrong, since it gave us the location of the Shadow-Copied assembles instead of the originals.
Instead you should use Assembly.CodeBase to fetch the base assembly code location. However, this is a (File) URL, so it’s necessary to extract the path from the URL. The new (C#) code looks like this:
var codeBaseUrl = new Uri(Assembly.GetExecutingAssembly().CodeBase);
var codeBasePath = Uri.UnescapeDataString(codeBaseUrl.AbsolutePath);
var dirPath = Path.GetDirectoryName(codeBasePath);
return Path.Combine(dirPath, relativePath);
…where relativePath is the path relative to the Bin\ directory.
After a bit of search I found the solution here: https://msdn.microsoft.com/en-us/library/ms182475.aspx.
Particularly, the first step has been enough for me:
If they are specific to one test project, include them as content files in the Visual Studio test project. Select them in Solution Explorer and set the Copy to Output property to Copy if Newer.
associated to the following code:
var filename = "./Resources/fake.pdf"
File.OpenRead(filename)
As of .NET5 (perhaps earlier), CodeBase no longer works, so the solution is now to copy all of your files to the bin directory first and use this as your known location.
What makes this OK now, which was always a total pain in the past, is that you can copy a directory to your bin folder easily from your csproj file.
<ItemGroup>
<None
Include="TestFiles\**"
CopyToOutputDirectory="PreserveNewest"
LinkBase="TestFiles\" />
</ItemGroup>
Where TestFiles is in the root of your project folder. Now you can access these files with the following helper method.
public static class TestUtils
{
public static string GetTestPath(string relativePath)
{
var codeBaseUrl = new Uri(Assembly.GetExecutingAssembly().Location);
var codeBasePath = Uri.UnescapeDataString(codeBaseUrl.AbsolutePath);
var dirPath = Path.GetDirectoryName(codeBasePath);
return Path.Combine(dirPath, "TestFiles", relativePath);
}
}
I am running .Net Core 1.0 on Mac. Assembly.GetExecutingAssembly is unavailable. I use the following code instead:
var location = typeof(YourClassName).GetTypeInfo().Assembly.Location;
var dirPath = Path.GetDirectoryName(location);
return Path.Combine(dirPath, relativePath);
relativePath is the path relative to the directory of your DLL.
After struggling with trying to identify directories and copy files (there seem to be multiple directories in play and it's hard to pin down the right one), I found that you can just turn off shadow copying; at that point the unit tests reference content in the bin/{configuration} folder again.
To do so, follow the directions here. It seems to be as simple as setting shadowCopy to false:
{
"shadowCopy": false
}
I'm unclear whether this setting has any adverse interactions with other settings (e.g. appDomain). If anyone knows, your comments would be welcome.

what's the purpose of *.vsmdi? Do I need to source control it?

What's the purose of .vsmdi file? Do I need to check into the source control system?
The VSMDI file is created by Visual Studio when you create a test project for the first time. It contains a list of all tests that Visual Studio can find in your solution assemblies and allows you to divide your tests into so-called test lists. These test lists can be used to categorize your tests and let you select a subset of tests for execution.
You can use this mechanism for running sub-selections. However, you can also (freely) assign multiple test categories to a test, which enables you to achieve the same, in a more flexible way. And with the known issues with VSMDI files, like uncontrolled duplication of these files and obsolete tests being listed with a warning icon, it might seem the better way to do things like this.
My overall suggestion is: check-in your default generated .vsmdi file. This will prevent Visual Studio from (re-)generating such files on your own and your team members systems when new test projects are added. Decide on usage of test lists or assigning categories to tests directly based on your usage experience. Test lists are easy to start with, but less suitable is you want to have flexibility for a large set of tests.
It's used for Testing in Visual Studio. If you don't do testing in Visual Studio, I wouldn't worry about it. But if you do, and you have hundreds of tests it might be worth keeping.

How to get MSTest to find my test data files?

I have a few tests that need to be fed with external data from excel files. The files are included in the test project, and in Visual Studio, I have edited the test settings file (Local.testsettings) to deploy the data files. This makes it work fine i VS.
We are, however, also running continous integration with TeamCity, and in TeamCity this doesn't work. My data files are unavailable to the test. Seems that the tests are run from a temporary folder named "C:\TeamCity\buildAgent\temp\buildTmp\ciuser_AS40VS6 2009-12-11 09_40_17\Out", and the data files are not copied there.
I have tried changing the build action for the data files to "Resource" and setting copy to output dir to "Always", but that didn't help.
Does anyone know how to make this work?
I am running Visual Studio 2010 beta 2 and TeamCity 4.5.5, which is why I'm running MSTest in the first place, and not NUnit...
I get round this by adding my data files (in my case usually XML) as embedded resources and I extract them from the test assembly.
[TestInitialize]
public void InitializeTests()
{
var asm = Assembly.GetExecutingAssembly();
this.doc = new XmlDocument();
this.doc.Load(asm.GetManifestResourceStream("TestAssembly.File.xml"));
}
This post answers this question: MSTest copy file to test run folder
The accepted answer is technically correct. However, from my experience, I find that the embedding files as resources requires an additional step of remembering to set the property "Embedded Resource". This becomes a challenge when you have a large number of data files. Also, with increasing number of data files, the size of the unit test assembly keeps growing . In my case, I had over 500MB of test data files and packing all them into the assembly was not a good idea.
What is the alternative?
Let the data files remain as they are. Do not use DeploymentItemAttribute, do not use embedded resources. Please refer my proposed solution How do I make a data file available to unit tests?

Resources