My unit tests have dependencies which are from a pool of dlls in a directory outside of the project. We have the policy to generally place links to these dlls (from the pool) in a directory called "System" inside the project directory and set their CopyToOutputDirectory to PreserveNewest. Afterwards, in the project we add a reference to the dll in the pool but set it to be not a private copy which means it will not be output to the build dir. In the InitAssembly() method we have setup that the runtime searches also in the AppDomain.CurrentDomain.BaseDirectory, "System") by the means of AppDomain.CurrentDomain.AssemblyResolve.
[TestClass]
public class SomeUnitTest : OurUnitTestBase
{
[AssemblyInitialize]
public static void AssemblyInit(TestContext context)
{
Debug.WriteLine("in AssemblyInit");
base.InitAssembly();
}
}
This works nice when running our code from inside VisualStudio and also when running the unit tests from inside VisualStudio's Test-Explorer (vstest.executionengine.x86.exe). However MSTest fails to find the dlls in the System subdirectory when run from the following batch script which executes a coverage analysis with opencover:
REM Bring dev tools into the PATH.
call "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\VsDevCmd.bat"
mkdir .\CoverageAnalysisReport
REM Run unit tests through OpenCover
REM This allows OpenCover to gather code coverage results
.\..\..\Tools\CoverageAnalysis\opencover\OpenCover.Console.exe^
-register:user^
-target:MSTest.exe^
-targetargs:"/noresults /noisolation /testcontainer:..\bin\Debug\OurUnitTests.dll"^
-filter:+[*]*^
-output:.\CoverageAnalysisReport\output.xml
REM Generate the report
.\..\..\Tools\CoverageAnalysis\ReportGenerator\bin\ReportGenerator.exe^
-reports:.\CoverageAnalysisReport\output.xml^
-targetdir:.\CoverageAnalysisReport^
-reporttypes:HTML^
-filters:-OurUnitTests*
REM Open the report
start .\CoverageAnalysisReport\index.htm
The resulting log file says, that none of the unit tests could be run and failed therefore. Furthermore it states
Warning: The assembly "BaseLib" as a direct or indirect dependency of the test container "C:\MyProject\bin\debug\ourunittests.dll" was not found.
Yet I know [AssemblyInitialize] was called because a System.Windows.Form.MessageBox.Show(..) I placed there temporarily actually got displayed during the execution of the bat-file.
Is there a way to tell MSTest to search also for dlls in the System subdirectory also?
I solved it by placing the following as the content of a file named Local.testsettings in the same directory like the batch file (in which I modified the line -targetargs:"/testcontainer:\"..\bin\x86\Debug\OurUnitTests.dll\" /testSettings:\".\Local.testsettings\"):
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Lokal" id="2fa4344c-1f2f-4a04-86f3-41d223b10333" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Description>Dies sind die standardmäßigen Testeinstellungen für einen lokalen Testlauf.</Description>
<Deployment>
<DeploymentItem filename="..\bin\Debug\System\" />
</Deployment>
<Execution>
<TestTypeSpecific>
<UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
<AssemblyResolution>
<TestDirectory useLoadContext="true" />
</AssemblyResolution>
</UnitTestRunConfig>
</TestTypeSpecific>
<AgentRule name="Execution Agents">
</AgentRule>
</Execution>
</TestSettings>
Related
I have tests which create local files with results. Can I define in the .testsetting file to pull these result files from the test agent, once the tests are finished?
I know that copying files in the other direction is possible, that is
<TestSettings name="BAT-SOME-NAME" id="111111ce-f87a-22222-85b0-cea1111111">
<Deployment>
<DeploymentItem filename="..\bin\Release\x64\MyFile.xml" />
</Deployment>
</TestSettings>
will copy MyFile.xml to the test agent before starting the tests. How can I define to pull MyCustomOutput.xml after the test are finished?
The .testsettings file has a section for set-up and clean-up scripts, ie batch command or .exe files etc. You could put suitable remote copy command into the clean-up script.
I'm trying to organize my unmanaged .dll dependencies in my Visual Studio solution but unless the DLLs are strewn about the top level of my solution they do not get built into the application's directory and then the application fails to run. I have done a lot of Googling on the subject and there seems to be a solution in the form of an app.config setting:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<runtime>
<assemblybinding xmlns="urn:schemas-microsoft-com:asm.v1">
<probing privatepath="lib" />
</assemblybinding>
</runtime>
</configuration>
I have a folder in my solution called "lib" and Copy Local is set to true for all files in the folder but the application is still unable to run. How can I keep my Visual Studio solution organized without breaking my application?
The only way I have found to solve this issue is through windows .bat files.
If there is a way to encode extra search paths into an unmanaged executable, I would like to know!
Here is an example of what I do:
SET PATH=$~dp0plugins\MY_PLUGIN\Debug;%PATH%
vcvars32.bat
devenv build\MY_PROJECT.sln
The first line adds the path of where I want visual studio to find my dll's in.
The second line ensures that the 'right' visual studio will open up. IE: if I have my environment set for VS2010, I dont want VS2012 to open up by default.
The last line invokes the IDE to open up my sln file.
The visual studio that opens up now has a path to the Debug folder of one of my plugins. When I debug my application, the correct dll will be found, loaded, and debugged with no issue - I verify this under the Debug->Windows->Modules panel.
This only works for the Debug configuration. If I want to run the release configuration, I change the .bat file to say 'Release', close Visual Studio and re-run the .bat file.
I apply this same pattern to running my executable. Set up the environment of where I want my dll's to come from and then execute my executable:
SET PATH=$~dp0plugins\Debug;%PATH%
... Repeat for other dll locations
"%~dp0Debug\MY_EXECUTABLE.exe" %*
SS64 is an invaluable resource to figure out what you can do in .bat files
I did quite a lot of research on the web and tried a few settings, but I couldn't reproduce the behavior of running MsTest in Visual Studio 2012 on the command line.
Our solution consists of many projects that build to the same bin folder residing at the solution level (e.g. C:\MySolution\bin) - this is the code-under-test (CUT). The tests are grouped in a separate project that resides in its own solution and builds in its own bin folder (e.g. C:\MySolution\Tests\bin). There are really a lot of plugins thus we want MsTest to reference the CUT bin folder when running the test intead of copying everything to the TestResults folder. We did achieve this in Visual Studio 2012 by editing the .testrunconfig and specifying ".\bin" as "root folder for the assemblies to be loaded" (in tab "Unit Test" when editing the testrunconfig). So we can load the test solution in VS2012 and run the tests there without having to copy the bin folder contents to the TestResults directory.
Now I wanted to create a .bat file that would run MsTest the same way as in VS2012 so that we can omit launching Visual Studio just for running the tests. I'm now working on how to execute MsTest on the command line, but have been quite frustrated. This is what I tried (command executed on the solution level in a VS command prompt):
MsTest /testcontainer:Tests\bin\Tests.dll
This did not work at all, it couldn't even find the referenced dlls that Tests.dll need to run. So I re-used the configuration and ran it again:
MsTest /runconfig:LocalTestRun.testrunconfig /testcontainer:Tests\bin\Tests.dll
Still it did not work. It could start the tests, but all of them failed. I got a lot of warnings of the kind
Warning: Test Run deployment issue: The assembly or module '....' directly or indirectly referenced by the test container 'C:\MySolution\Tests\bin\tests.dll' was not found.
and in the end it said:
The configured application base directory 'C:\MySolution\TestResults\User_Machine 2013-07-28 13_16_59\Out\bin' does not exist. The test directory will be used instead.
When I changed the option applicationBaseDirectory in the testrunconfig to an absolute path (C:\MySolution\bin), it worked. Still I get many warnings such as:
Warning: Test Run deployment issue: The assembly or module '....' directly or indirectly referenced by the test container 'C:\MySolution\Tests\bin\tests.dll' was not found.
But anyway, it's not really a feasible solution to specify an absolute path. How can I run MsTest on the command line with a different, but relative assembly base directory?
My LocalTestRun.testrunconfig is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Local Test Run" id="...." xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Description>This is a default test run configuration for a local test run.</Description>
<Deployment>
<DeploymentItem filename="Tests\....\Resources\" />
</Deployment>
<Execution hostProcessPlatform="MSIL">
<TestTypeSpecific>
<UnitTestRunConfig testTypeId="....">
<AssemblyResolution applicationBaseDirectory=".\bin">
<TestDirectory useLoadContext="true" />
</AssemblyResolution>
</UnitTestRunConfig>
<WebTestRunConfiguration testTypeId="....">
<Browser name="Internet Explorer 7.0">
<Headers>
<Header name="User-Agent" value="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" />
<Header name="Accept" value="*/*" />
<Header name="Accept-Language" value="{{$IEAcceptLanguage}}" />
<Header name="Accept-Encoding" value="GZIP" />
</Headers>
</Browser>
</WebTestRunConfiguration>
</TestTypeSpecific>
<AgentRule name="LocalMachineDefaultRole">
</AgentRule>
</Execution>
</TestSettings>
After more searching we changed to use the test console runner that is included with VS2012:
VSTest.Console.exe Tests\bin\Tests.dll /Framework:framework40 /Settings:LocalTestRun.testrunconfig
This works with a relative path as applicationBaseDirectory.
This is due to an MSTest bug that sets the current directory to its own working directory, rather than the test project's bin (or deployment) folder. The workaround is to execute the following code in the constructor of your test class:
Environment.CurrentDirectory(AppDomain.CurrentDomain.BaseDirectory);
I got the idea from http://www.ademiller.com/blogs/tech/2008/01/gotchas-mstest-appdomain-changes-in-vs-2008/; however, note that, in my case at least, it required setting Environment.CurrentDirectory, rather than the reverse, as suggested in the article.
I'm trying to export visual studio code coverage files (data.coverage) into xml as described in this blog post from the code analysis team. I've moved the code example in that post into a custom MSBuild task. My custom task references the Microsoft.VisualStudio.Coverage.Analysis.dll located in the PrivateAssemblies folder of Visual Studio.
Right off the bat, trying to load the code coverage file throws an code analysis typed exception ImageNotFoundException, stating that the "Image file fully-qualified-file-path-to-dll could not be found."
// the following line throws an exception
CoverageInfo current =
CoverageInfo.CreateFromFile( "c:\path\testresults\x\y\z\data.coverage");
The path is fully qualified and the DLL it refers to does exist. My testsettings has this file listed as the assembly to instrument and the "Instrument in place" checkbox is set. I can view code coverage within Visual Studio, so I know coverage is working.
I'm running my MSBuild script from the Visual Studio command line. It looks like this:
<Project ToolsVersion="4.0" DefaultTargets="Default;"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<UsingTask TaskName="CustomTasks.MergeCoverageTask"
AssemblyFile="CustomTasks.dll"
/>
<Target Name="Default">
<ItemGroup>
<CoverageFiles Include="**\data.coverage" />
</ItemGroup>
<MergeCoverageTask
CoverageFiles="#(CoverageFiles)"
OutputFile="output.xml"
/>
</Target>
</Project>
Can anyone suggest what I need to do to get this working correctly?
5 hours later and this is a tumbleweed. I found some additional detail here, which helped get me further down the path.
In order for this to work, you need to include a few additional files alongside the custom task and supply folder locations for the pdb's and instrumented dll's.
Regarding additional files, you need the following:
The custom build task must reference Microsoft.VisualStudio.Coverage.Analysis.dll
Your bin folder must contain the following additional files:
Microsoft.VisualStudio.Coverage.Symbols.dll
dbghelp.dll
(If you don't have visual studio installed, you must perform regsvr32.exe on msdia100.dll)
Regarding paths to assemblies and symbols, the CreateFromFile method takes a collection of folders to search. What seems really strange is that error complains about not being able to locate missing instrumented assemblies, and it specifies the full path..
Image file c:\project\output\Assembly.dll could not be found.
...but if you specify that path, it doesn't work.
CoverageInfo current =
CoverageInfo.CreateFromFile( "c:\project\testresults\x\In\data.coverage",
new string[] { "c:\project\output" },
new string[] { "c:\project\output" });
However, changing the path to be the folder of the TestResults output works fine:
CoverageInfo current =
CoverageInfo.CreateFromFile( "c:\project\testresults\x\In\data.coverage",
new string[] { "c:\project\testresults\x\Out" },
new string[] { "c:\project\testresults\x\Out" });
I question whether "instrument in place" really means in that folder, or instrument and copy to the MS Test run folder.
Well dear SO folk, if you're reading this, you get a cookie.
I've got a test which requires an XML file to be read in and then parsed. How can I have this file copied into the test run folder each time?
The XML file is set to "Copy if newer" and a compile mode of "none" (since it's not really a compile-able thing)
use a DeploymentItem attribute
using System;
using System.IO;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using CarMaker;
namespace DeploymentTest
{
[TestClass]
public class UnitTest1
{
[TestMethod()]
[DeploymentItem("testFile1.xml")]
public void ConstructorTest()
{
string file = "testFile1.xml";
Assert.IsTrue(File.Exists(file), "deployment failed: " + file +
" did not get deployed");
}
}
}
It seems that if you provide a TestSettings file for the Solution then you can uncheck the "Enable deployment" option and stop mstest from trying to run from the ...TestResults\...\out folder where it doesn't copy your extra files (unless you make them a deployment option).
This is also useful if you depend on the extra files being in a preserved folder structure because Deployment items all seem to be copied directly (flat) into the temporary run folder (out) if you use the Deployment, Add Folder option in the TestSettings (answers above suggest you can keep the structure if you add each item as its own DeploymentItem).
For me it worked fine running tests directly in Visual Studio (i.e. my extra files in their structure were found and used by tests) because I had created a TestSettings file for another reason long ago (which has Enable deployment unchecked), but not when TeamCity ran mstest to run tests because I hadn't specified that the TestSettings file should be used.
To create a TestSettings file in Visual Studio, right click on the Solution and choose New Item, and select the TestSettings template. To use the TestSettings file at the command prompt of mstest.exe add the option, /testsettings:C:\Src\mySolution\myProject\local.testsettings (or add as an extra command line option in TeamCity with appropriate path)
The Preet answer is used to deploy items for a single test. If you want to do it at solution level, use the .testrunconfig settings.
Best solution to me is using testsettings, especially if multiple tests need the same datafiles.
First create a testsettings file, and add the deployment items you need (file or folder name):
<TestSettings name="Local" id="00ebe0c6-7b64-49c0-80a5-09796270f111" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Description>These are default test settings for a local test run.</Description>
<Deployment>
<DeploymentItem filename="Folder1\TestScripts\test.xml" outputDirectory="TestScripts"/>
<DeploymentItem filename="Folder2\TestData\" outputDirectory="TestData"/>
</Deployment>
<...../>
Running in visual studio, use "select Test Settings File" from "Test\Test Settings" menu to select new testsettings
Running mstest, use the /testsettings parameter to have mstest use your testsettings.
You can define DeploymentItem in a class that holds a method with AssemblyInitialize attribute. Then you're sure that the files are copied regardless of which test you run.
Unfortunately DeploymentItem attribute is executed only on classes which contain tests you're running. So if you have 10 test classes which use the same set of files, you have to add the attribute to all of them.
Also found out that changes in *.testsettings files are not automatically refreshed in Visual Studio. Therefore after adding files / folders into deployment in testsettings, you have to reopen solution file and then run the tests.
In Visual Studio 2012, vstest.console.exe (the built-in test runner) runs with the output dir as the current path. This means that you only need to include the items in your solution with the 'Copy always' or 'Copy if newer' property for them to be used by your test. You don't need the DeploymentItem attribute for the general case. The same applies when running vstest.console.exe from the command line inside your output/test directory.
There are some cases where a separate folder is used, one of them being when you are using the DeploymentItem attribute. See here for more information.