I started using Ordered tests recently in VS 2010. I have created a folder under my test-suite and dragged the tests (required tests in a flow) from test list editor to the ordered test list. I am able to run the ordered test from VS successfully. But when I run the ordered test using MSTest, ordered test gets error (all of its tests gets not executed). When I opened the Ordered test file in notepad, I see everything properly configured like storage path as shown below
*<?xml version="1.0" encoding="UTF-8"?>
<OrderedTest name="MyOrderedTest" storage="OrderedTests\MyOrderedTest.orderedtest" id="afadbaf6-7915-426d-932c-788fc27c7a8f" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<TestLinks>
<TestLink id="c2367ab7-ab71-bdaf-d039-4d55ea87e962" name="TestOne" storage="TestSuiteName.dll" type="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestElement, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
<TestLink id="cc156aeb-ed8e-12d6-c9ad-24fadc2ca54b" name="TestTwo" storage="TestSuiteName.dll" type="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestElement, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
<TestLink id="caa09202-b582-e53c-cae9-7fe8b402b4a2" name="TestThree" storage="TestSuiteName.dll" type="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestElement, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
</TestLinks>
</OrderedTest>*
When I ran a single test 'TestOne' using MSTest, it runs successfully.
Is there anyother thing that I need to configure/change??
Thanks in Advance.
Sham_
Did I understand you correctly: you have stored your ordered test in a sub-folder?
If so, just move your ordered test in to the root folder (do not use a sub-folder for storing them).
I had been facing the same problem using ordered tests for test automation. When running them from Microsoft Test Manager I got the error
"Cannot find the test ‘XXX’ with storage ‘..\bin\debug\YYY.dll"
According to this message Test Agent (the component that runs the tests) is looking for them in the root folder and not in a sub-folder I've stored them in.
Moving the ordered tests to the root folder of my solution solved the problem.
It seems to be a bug.
PS: You can check in the Output window why the tests were not executed.
Even with 2012 version this issue exists . Moving the ordered test to root folder solved the issue for me as well.
Related
VSTest, which is what TFS 2015 uses for unit-tests, wants to ignore application config/settings (app.config) by default. VSTest takes an optional "runsettings" configuration file, but it doesn't accommodate any information about an app.config file. Now, from the runsettings file, you can tell it to drop into "legacy" mode:
<RunSettings>
<MSTest>
<ForcedLegacyMode>true</ForcedLegacyMode>
</MSTest>
</RunSettings>
...and this will allow it to use the app.config file in the same directory (these are assemblies/libraries, so the configuration file is just named "app.config" rather than "application.exe.config"). However, it will not find included files. For example, I have a separate connection-strings file. It's included from the top of the app.config:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<!-- For more information on Entity Framework configuration, visit http://go.microsoft.com/fwlink/?LinkID=237468 -->
<section name="entityFramework" type="System.Data.Entity.Internal.ConfigFile.EntityFrameworkSection, EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false" />
</configSections>
<connectionStrings configSource="connectionStrings.config" />
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
</startup>
I will get an error every time about how it can't find the file:
2016-09-21T18:36:12.2439565Z ##[error]Error Message:
2016-09-21T18:36:12.2439565Z ##[error] Test method Assembly.Project.Class threw exception:
2016-09-21T18:36:12.2595841Z ##[error]System.Configuration.ConfigurationErrorsException: Unable to open configSource file 'connectionStrings.config'. (D:\Agents\A3\_work\464b36816\TestResults\Definition_Host 2016-09-21 14_36_09\Out\Assembly.DLL.config line 7)
2016-09-21T18:36:12.2595841Z ##[error]Stack Trace:
Since it's only allowed to be relative (otherwise, you'll get another error), I've tried prepending ".\" (which seems like it works in other situations) and well as referencing the next few levels of directories above it, but it will not find the referenced connection-strings file.
Okay. So I wanted to inspect the execution logic in order to figure out what the working directory was. The TFS 2015 task that is initiating the tests is called VSTest.ps1. It refers to a cmdlet called "Invoke-PSTest" in order to call "vstest.console.exe" to actually run the tests. This exists absolutely nowhere on the disk as it's own file or in any psd1/psm1 files, so it must be provided by a binary assembly.
However, I'm also apparently avoiding providing a data-source/connection-string to the unit-tests the way that I am supposed to. After all, I'm having to use "legacy" mode in order to get VSTest to use my configuration file.
So, what is the proper way to provide application-configuration (such as a connection-string) to unit-tests? After-all, sometimes you actually need an external resource in order to properly test.
I'm working on a web site (not web application) project in Visual Studio 2013 and have intermittent build issues. The build output shows each directory building (e.g. "Building directory '/User/'."). However, more often than not, on different directories, during different build attempts, I receive the following error (note the filename changes):
The file name 'C:\Users\xxxx\AppData\Local\Temp\2\Temporary ASP.NET Files\root\25143ee9\2d39f4f7\yhzrc1w3.pdb' was already in the collection. Parameter name: fileName
And the following build output:
: Build (web): The file name 'C:\Users\xxxx\AppData\Local\Temp\2\Temporary ASP.NET Files\root\25143ee9\2d39f4f7\yhzrc1w3.pdb' was already in the collection.
I have tried deleting everything from the temp directories, restarting visual studio, bypassing Visual Studio (building with aspnet_compiler), and doing a rain dance. However, no matter what I try builds intermittently fail.
Is there any way to see more granular output from the build process in Visual Studio? I've already tried setting the build output to detailed ("Tools"-->"Projects and Solutions"-->"Build and Run"-->"MSBuild project build output verbosity:"). Any ideas would be greatly appreciated!
Thank you!
I ended up finding out what was causing the problem.
I had upgraded from ReportViewer 10 to ReportViewer 11 and the following line was added to the web.config:
<buildProviders>
<add extension=".rdlc" type="Microsoft.Reporting.RdlBuildProvider, Microsoft.ReportViewer.WebForms, Version=11.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91" />
</buildProviders>
This was causing the intermittent build failures. I removed the buildProvider tag and it builds and deploys just fine.
I want to add multiple Ms unit test project in single solution in VS 2010 and want to execute all projects test case in single shot in Bamboo CI.
In Bamboo i am able to test Single project by using project dll but not able to execute Test written in multiple project .
I tried on google and found .vsmdi file will do that functionality but not crack how to all test cases add in to vsmdi file .
Please help me.
The Vsmdi File is look like this :
<?xml version="1.0" encoding="UTF-8"?>
<TestLists xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<TestList name="Lists of Tests" id="8c43106b-9dc1-4907-a29f-aa66a61bf5b6">
<RunConfiguration id="68cd1a40-fbe7-4a52-bc91-47f0ca5ea70a" name="Local" storage="local.testsettings" type="Microsoft.VisualStudio.TestTools.Common.TestRunConfiguration, Microsoft.VisualStudio.QualityTools.Common, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
</TestList>
</TestLists>
I did quite a lot of research on the web and tried a few settings, but I couldn't reproduce the behavior of running MsTest in Visual Studio 2012 on the command line.
Our solution consists of many projects that build to the same bin folder residing at the solution level (e.g. C:\MySolution\bin) - this is the code-under-test (CUT). The tests are grouped in a separate project that resides in its own solution and builds in its own bin folder (e.g. C:\MySolution\Tests\bin). There are really a lot of plugins thus we want MsTest to reference the CUT bin folder when running the test intead of copying everything to the TestResults folder. We did achieve this in Visual Studio 2012 by editing the .testrunconfig and specifying ".\bin" as "root folder for the assemblies to be loaded" (in tab "Unit Test" when editing the testrunconfig). So we can load the test solution in VS2012 and run the tests there without having to copy the bin folder contents to the TestResults directory.
Now I wanted to create a .bat file that would run MsTest the same way as in VS2012 so that we can omit launching Visual Studio just for running the tests. I'm now working on how to execute MsTest on the command line, but have been quite frustrated. This is what I tried (command executed on the solution level in a VS command prompt):
MsTest /testcontainer:Tests\bin\Tests.dll
This did not work at all, it couldn't even find the referenced dlls that Tests.dll need to run. So I re-used the configuration and ran it again:
MsTest /runconfig:LocalTestRun.testrunconfig /testcontainer:Tests\bin\Tests.dll
Still it did not work. It could start the tests, but all of them failed. I got a lot of warnings of the kind
Warning: Test Run deployment issue: The assembly or module '....' directly or indirectly referenced by the test container 'C:\MySolution\Tests\bin\tests.dll' was not found.
and in the end it said:
The configured application base directory 'C:\MySolution\TestResults\User_Machine 2013-07-28 13_16_59\Out\bin' does not exist. The test directory will be used instead.
When I changed the option applicationBaseDirectory in the testrunconfig to an absolute path (C:\MySolution\bin), it worked. Still I get many warnings such as:
Warning: Test Run deployment issue: The assembly or module '....' directly or indirectly referenced by the test container 'C:\MySolution\Tests\bin\tests.dll' was not found.
But anyway, it's not really a feasible solution to specify an absolute path. How can I run MsTest on the command line with a different, but relative assembly base directory?
My LocalTestRun.testrunconfig is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Local Test Run" id="...." xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Description>This is a default test run configuration for a local test run.</Description>
<Deployment>
<DeploymentItem filename="Tests\....\Resources\" />
</Deployment>
<Execution hostProcessPlatform="MSIL">
<TestTypeSpecific>
<UnitTestRunConfig testTypeId="....">
<AssemblyResolution applicationBaseDirectory=".\bin">
<TestDirectory useLoadContext="true" />
</AssemblyResolution>
</UnitTestRunConfig>
<WebTestRunConfiguration testTypeId="....">
<Browser name="Internet Explorer 7.0">
<Headers>
<Header name="User-Agent" value="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" />
<Header name="Accept" value="*/*" />
<Header name="Accept-Language" value="{{$IEAcceptLanguage}}" />
<Header name="Accept-Encoding" value="GZIP" />
</Headers>
</Browser>
</WebTestRunConfiguration>
</TestTypeSpecific>
<AgentRule name="LocalMachineDefaultRole">
</AgentRule>
</Execution>
</TestSettings>
After more searching we changed to use the test console runner that is included with VS2012:
VSTest.Console.exe Tests\bin\Tests.dll /Framework:framework40 /Settings:LocalTestRun.testrunconfig
This works with a relative path as applicationBaseDirectory.
This is due to an MSTest bug that sets the current directory to its own working directory, rather than the test project's bin (or deployment) folder. The workaround is to execute the following code in the constructor of your test class:
Environment.CurrentDirectory(AppDomain.CurrentDomain.BaseDirectory);
I got the idea from http://www.ademiller.com/blogs/tech/2008/01/gotchas-mstest-appdomain-changes-in-vs-2008/; however, note that, in my case at least, it required setting Environment.CurrentDirectory, rather than the reverse, as suggested in the article.
I am trying to use Gallio (v3.1)/MbUnit/NCover to run a unit test in my C# code, as part of the build process for my continuous integration system.
I can get Gallio.Echo.exe to execute the tests and output an XML file (albeit it does seem to be checking all .dll files in the folder == approx. 6.5MB .xml file!!), but when I try to get NCover to link in also, it goes bang.
THEN: I tried to use the NAnt task using instructions from here, such as:
<gallio result-property="testrunner.exit-code"
application-base-directory="bin/debug"
runner-type="NCover"
failonerror="false"
report-name-format="gallio-MyTestProject"
report-types="xml"
report-directory="bin/debug">
<runner-property value="NCoverArguments='//q //ea CoverageExcludeAttribute //a MyTestProject.dll'" />
<runner-property value="NCoverCoverageFile='coverage-MyTestProject.xml'" />
<assemblies>
<include name="bin/debug" />
</assemblies>
</gallio>
but I get the following error on my command-line:
Element Required! There must be a least one 'files' element for <gallio ... />.
I have tried to specify the .dll file that I'd like to check, but it still comes up with this message. Any suggestions are most appreciated!
<assemblies> has been changed to <files>