We are using VSTS to perform Continuous Integration utilizing a build template that was created to deploy a website to a Dev machine.
This worked fine but then we wanted to add in CodedUI tests to the build template in order to run the tests as part of the build step.
So I added a Visual Studio Test step to the build and added in the test by referencing the test plan and test suite it was associated with.
I made sure the codedUI project was built within the same solution as the website in order to make sure the binaries were built with it.
The problem I'm getting is that when the build runs it brings back errors stating that it is missing the relevant references for the code to work such as:
QualityTools.CodedUITestFramework
QualityTools.Common
TestTools.UITesting
etc.
However these reference are included in the project and "Copy Local" is set to true.
Does anyone have any idea what could be going wrong?
UPDATE
New error received:
MSTestAdapter failed to discover tests in class
'Email_Regression.CodedUITest1' of assembly
'C:\agent_work\4\s\Microservices\Email\Email
Regression\obj\Release\Email Regression.dll'. Reason Could not load
file or assembly
'Microsoft.VisualStudio.QualityTools.CodedUITestFramework,
Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or
one of its dependencies. The system cannot find the file specified..
No test is available in C:\agent_work\4\s\Microservices\Email\Email
Regression\bin\Release\Email Regression.dll
C:\agent_work\4\s\Microservices\Email\Email
Regression\obj\Release\Email Regression.dll. Make sure that test
discoverer & executors are registered and platform & framework version
settings are appropriate and try again.
After talking to the Microsoft help team it appears that you need to have Visual Studio Enterprise edition installed on the machine you wish to run the codedUI tests on.
Update
After discussions with Microsoft and highlighting this being a rubbish solution they are now going to deliver the dll's as part of the new vsTestPlatform installer. I'm not saying this is down to my complaints, just coincidental timing.
I upgraded to Visual Studio 2013 after using VS 2012.
In VS 2012, when created new Test Project and add files to be "Copy Always" the visual studio copied them into bin/debug and while running the current directory(Environment.CurrentDirectory) was "bin/debug".
In VS 2013 the current directory is "TestResults/something+guid" and VS 2013 is not copying the files to this folder so an "file not found" exception thrown.
How do I change back the current directory to bin/debug in VS2013 to be like VS 2012?
Thanks!
The proper way to handle this scenario is to use DeploymentItemAttribute or test settings config to include the file(s) you want. It is this way because not all tests require the same files, different tests may require different configs entirely, or you may have several test runs and would need to inspect test artifacts to understand why one run failed and another succeeded (where the only difference was a referenced assembly, or a loose file.) Further, when run in a hosted environment (Team Foundation Server, for example) this same pattern is used on the agent server(s), writing a test to set current directory would fail when run on something like a TFS test agent.
As an aside, the path you see during a test is not actually VS2013, it's the MS Test Agent itself (a second process runs tests for sandboxing purposes, I beleive it has been this was since VS2010.)
It sounds like you have an additional problem, all but guaranteed to be one of the following. Without seeing the exception detail I'm making a best guess, these are in order of likelihood:
You have a loose file such as "test.txt" included in your test project and you have a DeploymentItemAttribute decorating your test method. But the "Copy to output folder" setting for "test.txt" is set to "Do not copy".
Changing this setting, rebuilding, then retesting should work.
You have a missing assembly reference, likely for a non-BCL assembly (or, more specifically, you're missing a reference to an assembly which is referenced by an assembly you are referencing.)
To fix this problem you should load fuslogvw.exe and use the log data to discover any missing assembly reference(s) to your test project.
You have referenced a native DLL or assembly which is compiled for a specific processor architecture (x86 vs x64 vs MSIL) and it cannot be loaded within the processor architecture which MS Test Agent is running.
The solution to this is to use an assembly with the correct processor architecture when running the test.
Between project references, "Copy to output directory" setting, and [DeploymentItem], your tests should be finding the file(s) they rely on.
Let me know if you need more info, if you're still having a problem i would suggest editing the question to include the exception detail (or at least the parts that matter, such as Type, Message and the first 5 lines of StackTrace.)
I am attempting to run SpecFlow tests from the Visual Studio 2010 Command Prompt, and I am getting a rather obtuse error message:
Unit Test Adapter threw exception:
Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information..
Some info about my VS2010 Project setup:
Windows 7 Enterprise, 64 bit (Version 6.1.7601 Service Pack 1 Build 7601)
Visual Studio 2010 Premium (v10.0.40219.1 SP1Rel)
Using Coded UI Tests
Using SpecFlow 1.9.0, which delegates to the CodedUI test API
MSTest
.NET v4.0.30319
Whole solution is compiling to 32 bit code (I have coworkers using XP still)
I have a post-build event that copies a few DLL files from my NuGet packages directory to the target dir:
copy $(SolutionDir)packages\SpecBind.1.2.1.71\lib\net45\SpecBind.dll $(TargetDir)
copy $(SolutionDir)packages\SpecBind.CodedUI.1.2.1.71\lib\net45\SpecBind.CodedUI.dll $(TargetDir)
Without this, mstest somehow couldn't load a number of SpecFlow assemblies.
Relevant Parts of Test Project's App.config
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="specFlow" type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow" />
<section name="specBind" type="SpecBind.Configuration.ConfigurationSectionHandler, SpecBind" />
</configSections>
<connectionStrings>
...
</connectionStrings>
<specFlow>
<!-- For additional details on SpecFlow configuration options see http://go.specflow.org/doc-config -->
<unitTestProvider name="MsTest" generatorProvider="Specflow.CodedUI.MsTestCodedUiGeneratorProvider, Specflow.CodedUI" runtimeProvider="TechTalk.SpecFlow.UnitTestProvider.MsTest2010RuntimeProvider, TechTalk.SpecFlow" />
<stepAssemblies>
<!-- This attribute is required in order to use StepArgument Transformation as described here;
https://github.com/marcusoftnet/SpecFlow.Assist.Dynamic/wiki/Step-argument-transformations -->
<stepAssembly assembly="SpecFlow.Assist.Dynamic" />
<stepAssembly assembly="SpecBind" />
</stepAssemblies>
</specFlow>
<specBind>
<browserFactory provider="SpecBind.CodedUI.CodedUIBrowserFactory, SpecBind.CodedUI" browserType="IE" />
<!-- For additional details on SpecBind configuration options see the project site. -->
</specBind>
</configuration>
The command I was using to start the tests:
C:\path\to\bin\Debug> mstest /testcontainer:MyTests.dll /test:SpecFlowFeatureName
Loading MyTests.dll
Starting Execution...
Results Top Level Tests
-------- ------------------
Failed ...
Failed ...
Failed ...
...
I've been searching high and low for a solution, and all I keep finding is references to VS2008 and disabling Code Coverage. I have VS2010 and Code Coverage is not enabled in my local test settings.
Windows Event Log Viewer
After rooting around in my Windows Event Log viewer, I finally came across more info (I've heard people complain they can't find a stack trace with this error -- look in your Event Viewer)
The description for Event ID 0 from source VSTTExecution cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
(QTAgent32.exe, PID 6920, Thread 213) Unit Test Adapter threw exception: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at System.Reflection.RuntimeModule.GetTypes()
at System.Reflection.Assembly.GetTypes()
at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.GetAssemblyInfo(Assembly assembly)
at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.GetTypeInfo(Type type, Boolean checkAlreadyExaminedType)
at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.ResolveMethods()
at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.Initialize(UnitTestResult result, UnitTestRunner runner, ConsoleOutputRedirector redirector)
at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestRunner.Run(UnitTestElement test, ITestContext testContext, Boolean isLoadTest, Boolean useMultipleCpus)
the message resource is present but the message is not found in the string/message table
I don't know how to resolve this error. SpecFlow tests run fine from inside Visual Studio when I right-click in a .feature file loaded in the editor and choose "Run SpecFlow Scenarios..." but I can't run them from the command line.
Any ideas, or additional information I could post?
As a side note, this stemmed from my earlier question: How do you run SpecFlow scenarios from the command line using MSTest?
Update #1
I did a little more poking around, and found this blog post: MSTest and 64bit. I checked my local test settings, and it is forcing the tests to run in 32 bit.
Now using this command:
mstest /testcontainer:MyTests.dll /testsettings:"..\..\..\Local.testsettings"
Error message:
Test method Blah.Dee.Blah threw exception:
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestTools.UITest.Extension, Version=11.0.0.0, Culture=neutral, PublicKeyToken=...' or one of its dependencies. The system cannot find the file specified.WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
TestCleanup method Blah.Dee.Blah.ScenarioTearDown threw exception. BoDi.ObjectContainerException: BoDi.ObjectContainerException: Interface cannot be resolved: SpecBind.BrowserSupport.IBrowser.
Update #2
I removed the reference to Microsoft.VisualStudio.TestTools.UITest.Extension and added it back. Cleaned. Rebuilt. Ran the same command again. Still getting the same error.
For some reason it's trying to load the Microsoft.VisualStudio.TestTools.UITest.Extension assembly as Version 11, and I have Version 10 installed on my machine. I'm trying to track down where this version is being specified so I can change it.
Just in case anyone else comes across this, for me it was because code coverage was enabled. Disabling it resolved the issue:
http://blogs.msdn.com/b/danielvl/archive/2010/02/16/enable-code-coverage-in-visual-studio.aspx
Open the .testsettings file
Under "Test Settings", click "Data and Diagnostics"
Uncheck "Code Coverage"
Hit "Apply"
With the help of #jessehouwing's comment above, I've got this fixed. I was running into two problems.
The error I originally referenced was actually caused by mstest attempting to run the tests in a 64bit process (QTAgent.exe). I needed it to run as a 32 bit process because the project was compiled as 32 bit code on my 64 bit Windows 7 machine.
In Visual Studio 2010:
Click the "Test" menu -> "Edit Test Settings" -> "Local (local.testsettings)"
Click "Hosts" in the left hand panel
Make sure "Force tests to run in 32 bit process" is chosen in the "Run tests in 32 bit or 64 bit process" drop down. For me this was the default
In the Visual Studio Command Prompt, run the tests by referencing the local.testsettings file generated by Visual Studio:
mstest /testcontainer:foo.dll /testsettings:"path/to/solution/local.testsettings" /test:SpecFlowFeatureName
The second problem I had was my own fault. Earlier I was playing around with Coded UI tests and SpecFlow and had installed SpecBind, which threw a bunch of settings in App.config for my test project. This NuGet package is only for .NET 4.5, and we use .NET 4.0. I removed that NuGet package and settings from App.config, cleaned and rebuilt. This fixed this issue completely.
So in short, I:
Forced mstest to run in a 32 bit process, and referenced this local.testsettings file from the command line
Removed all references to SpecBind because I got a little NuGet-happy earlier and installed a package incompatible with my .NET runtime.
As you can see by looking at the assembly references required by these packages (I use Reflector for this, but IlSpy or JetBrains' DotPeek will work as well), you'll see that they depend on the v11 version, which is the version that ships with Visual Studio 2012. These can't be used in Visual Studio 2010.
Removing SpecBind will probably help, or finding a version that depends on an older version of CodedUI (you're probably looking for a much older version of SpecBind in this case).
As for your 32-bit issue, you should probably set your project's CPU configuration to AnyCPU, there is normally no need to fix it to x86, unless you're dependent on native methods or if you're referencing other assemblies that are specifically set to use x86 as their target architecture.
This is so troubling.....
I had Test Agent for Visual Studio 2015 installed on my test machine, since we were planning to use it - while on my local machine I had 2013. The tests ran so perfectly with MsBuild on local, that I couldn't understand what could have happened given that I always assumed Microsoft always maintained backward compatability.
MsBuild on test server failed to give any information as to what assemblies were missing (I added Newtonsoft.Json just because it complained initially, but after that I never saw the names of anything else missing). The error message was truncated, stacktrace and debugtrace options never worked...
Once I installed Test Agent for Visual Studio 2013 on the test machine, it ran like a charm.
I've just encountered a compile error I've not seen before. In a solution that contains Windows Workflow Foundation 3 projects, Visual Studio reports that it failed to compile the projects because it couldn't find an application-specific configuration file. The configuration file is NOT added to any of the projects. It is looked for only at runtime by the workflows. Other solutions using the same runtime config file compile without error. If I put the config file in the expected location then the workflow solution compiles without error.
The behavior suggests that when VS2010 is compiling WF3 projects that it is actually running the activities contained in the project at some point during the compile. Does VS actually run activities in WF projects during compile? Does it create activities? If it doesn't, why would the compile error be thrown for a file that isn't part of the project?
Actually I believe it suggests you have a compile time dependency on the config file in question on this one solution that you do not have in the others you mentioned.
Check if the error provides any details as to WHERE it has this compile time dependency and remove it if it is not needed. If there is nothing informing you or hinting where this may be, start digging into the various project properties in question and see if you have a build dependency on the config file.
I am following this guide to installing and using MSpec, but at the step where he runs MSpec for the first time, I get the following error:
Could not load file or assembly 'file:///[...]\Nehemiah\Nehemiah.Specs\bin\Debug\Nehemiah.Specs.dll' or one of its dependencies. This assembly is built by a runtime newer than the currently loaded runtime and cannot be loaded.
I have - to my knowledge - done everything more or less exactly like he did up to this step, except where differences arise because he's using VS2008 and I'm using VS2010, and everything has worked so far. The project Nehemijah.Specs (and the entire solution) builds without problem, both in Visual Studio and on my build server, and I can't find anything useful in Event Viewer (although I might not be looking in the right place here...)
What to do?
It appears you're compiling the Nehemiah.Specs.dll assembly against the .NET Framework 4. In order to run your specifications you need the .NET 4 build of MSpec which contains a console runner (mspec.exe) that is suitable for this version of the framework. You can find the a zip file containing the build on CodeBetter's CI server.
Other possibilities of running .NET 4 spec assemblies are
TestDriven.Net 3.0
ReSharper 5.0
Runners for both tools are also included in the distribution.