We are using VSTS to perform Continuous Integration utilizing a build template that was created to deploy a website to a Dev machine.
This worked fine but then we wanted to add in CodedUI tests to the build template in order to run the tests as part of the build step.
So I added a Visual Studio Test step to the build and added in the test by referencing the test plan and test suite it was associated with.
I made sure the codedUI project was built within the same solution as the website in order to make sure the binaries were built with it.
The problem I'm getting is that when the build runs it brings back errors stating that it is missing the relevant references for the code to work such as:
QualityTools.CodedUITestFramework
QualityTools.Common
TestTools.UITesting
etc.
However these reference are included in the project and "Copy Local" is set to true.
Does anyone have any idea what could be going wrong?
UPDATE
New error received:
MSTestAdapter failed to discover tests in class
'Email_Regression.CodedUITest1' of assembly
'C:\agent_work\4\s\Microservices\Email\Email
Regression\obj\Release\Email Regression.dll'. Reason Could not load
file or assembly
'Microsoft.VisualStudio.QualityTools.CodedUITestFramework,
Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or
one of its dependencies. The system cannot find the file specified..
No test is available in C:\agent_work\4\s\Microservices\Email\Email
Regression\bin\Release\Email Regression.dll
C:\agent_work\4\s\Microservices\Email\Email
Regression\obj\Release\Email Regression.dll. Make sure that test
discoverer & executors are registered and platform & framework version
settings are appropriate and try again.
After talking to the Microsoft help team it appears that you need to have Visual Studio Enterprise edition installed on the machine you wish to run the codedUI tests on.
Update
After discussions with Microsoft and highlighting this being a rubbish solution they are now going to deliver the dll's as part of the new vsTestPlatform installer. I'm not saying this is down to my complaints, just coincidental timing.
Related
I've been searching all morning for an answer online to this and I have tried a lot of the suggestions though all solutions seem to be for visual studio 2010 and I am running 2013 premium edition, I can't see anyone with the same problem.
I have premium VS 2013, I installed SpecFlow v1.9 through the extension manager and added the SpecFlow nuget package. I have created a CodedUI test project and added a feature file, some recorded codedui steps and a stepdef file. Nothing is complaining and it all looks like it should work. I have added :
<unitTestProvider name="MsTest"/>
to my App.config so I can run it from the test explorer in visual studio. I eventually want to run the tests via MTM but I will deal with that when I get this to work!
I have built and it is all happy so I go to test explorer and I can see my test so I right click, run the test. It fails with the following error, it does get to my Given step when I debug and falls over trying to open my application.
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestTools.UITest.WindowsStoreUtility, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
I can not find the reference above when I have looked to add it to my project references, there is a similar one but it breaks some of my other references when I add it. I also had a look at my registry files and the path it is referring to does not exist for me to edit it. From what I have read codedui doesn't work nicely with specflow without a dll file. All references to this fix seem to be for vs 2010 and require that I copy a dll to my specflow source folder. I installed specflow from visual studio so I don't have a program files folder for it so I came to a bit of a sticking point with that solution.
Does anyone have specflow working with visual studio 2013 and a codedui test project? Am I missing a set up step? Or is this genuinely to do with my registry files / references. The error is confusing as I don't see why it is trying to find that reference especially when I can't find that reference anywhere when I try to find it manually. I'd be interested to know if you have had this error and managed to resolve it or if you could explain your set up steps so I can check I did not miss anything.
Thanks!
I believe you might be missing references to some of the required dlls. I just finished testing a test created using a Specflow, codedUI test project on my MS Visual Studio 2013 ultimate and it worked just fine.
Here are the steps which I followed:
Created a class library project.
Added reference to Specflow 1.9 from Extensions and Updates.
Added reference to Specflow CodedUI Attribute Generator (Install-Package SpecFlow.CodedUI -Version 1.0.0.23).
Added references to required libraries required for CodedUI and Test Tools to the project. See below the references I added.
Microsoft.VisualStudio.QualityTools.CodedUITestFramework
Microsoft.VisualStudio.QualityTools.UnitTestFramework
Microsoft.VisualStudio.TestTools.UITest.Common
Microsoft.VisualStudio.TestTools.UITest.Extension
Microsoft.VisualStudio.TestTools.UITest.ExtensionUtilities
Microsoft.VisualStudio.TestTools.UITesting
Added a separate CodedUI project to solution and added references of this project to the class library project.
Create a new Specflow feature file with some test and create step for it in step definition and reference to the function you created or recorded in the CodedUI project in the step definition.
Run the Specflow test from Test Explorer. It should work fine.
Note: The above steps should work just fine if you will create CodedUI project instead of a Class library project.
Mark the test class with the CodedUITest attribute.
I had this exact error message even though I had all the .dll's referenced correctly.
The solution for me was to regenerate the feature files.
(right click project and select "regenerate feature files")
Leaving this answer in case it helps someone in future.
I came across the same issue today, not with specflow but 2013 build controller and agent and codedui tests.
The solution was to install VS2013 Update 5 https://www.visualstudio.com/downloads/download-visual-studio-vs
I had same problem, found the following solution works well:
http://blog.majcica.com/2015/05/07/getting-started-with-specflow-and-codedui/
I upgraded to Visual Studio 2013 after using VS 2012.
In VS 2012, when created new Test Project and add files to be "Copy Always" the visual studio copied them into bin/debug and while running the current directory(Environment.CurrentDirectory) was "bin/debug".
In VS 2013 the current directory is "TestResults/something+guid" and VS 2013 is not copying the files to this folder so an "file not found" exception thrown.
How do I change back the current directory to bin/debug in VS2013 to be like VS 2012?
Thanks!
The proper way to handle this scenario is to use DeploymentItemAttribute or test settings config to include the file(s) you want. It is this way because not all tests require the same files, different tests may require different configs entirely, or you may have several test runs and would need to inspect test artifacts to understand why one run failed and another succeeded (where the only difference was a referenced assembly, or a loose file.) Further, when run in a hosted environment (Team Foundation Server, for example) this same pattern is used on the agent server(s), writing a test to set current directory would fail when run on something like a TFS test agent.
As an aside, the path you see during a test is not actually VS2013, it's the MS Test Agent itself (a second process runs tests for sandboxing purposes, I beleive it has been this was since VS2010.)
It sounds like you have an additional problem, all but guaranteed to be one of the following. Without seeing the exception detail I'm making a best guess, these are in order of likelihood:
You have a loose file such as "test.txt" included in your test project and you have a DeploymentItemAttribute decorating your test method. But the "Copy to output folder" setting for "test.txt" is set to "Do not copy".
Changing this setting, rebuilding, then retesting should work.
You have a missing assembly reference, likely for a non-BCL assembly (or, more specifically, you're missing a reference to an assembly which is referenced by an assembly you are referencing.)
To fix this problem you should load fuslogvw.exe and use the log data to discover any missing assembly reference(s) to your test project.
You have referenced a native DLL or assembly which is compiled for a specific processor architecture (x86 vs x64 vs MSIL) and it cannot be loaded within the processor architecture which MS Test Agent is running.
The solution to this is to use an assembly with the correct processor architecture when running the test.
Between project references, "Copy to output directory" setting, and [DeploymentItem], your tests should be finding the file(s) they rely on.
Let me know if you need more info, if you're still having a problem i would suggest editing the question to include the exception detail (or at least the parts that matter, such as Type, Message and the first 5 lines of StackTrace.)
I started creating automated builds in TFS 2012 using a build server with VS2012 on it. Everything was running fine, but then we updated the build server to VS2013(still with TFS 2012). Now no units execute. I get the error:
Handle Test Run Exception
Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.Utilities,
Version=11.0.0.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
The system cannot find the file specified.
I can find that file on the build server but only version 12 under vs2013.
Im lost as to what to do as I dont know what is forcing the Test Runner to use version 11 or how to change it.
Is it possible that since the build controller is tfs 2012, it is somehow forcing the Test Runner in the build to use version 11 instead of version 12 on the build server? Any ideas how to resolve this?
You need to have VS Professional or later on your build server to run automated tests as a part of your build process.
Here since TFS is still at 2012 the test activities are referring the 11.0 dlls and installing VS2012 should help resolve this issue.
This is a known bug being tracked.
From http://msdn.microsoft.com/en-us/library/ms253138.aspx
To run unit tests, or coded UI tests, you must install Visual Studio
Professional or later
I am trying to build MSIs in a TFS Build by shelling out to DEVENV.exe (since MSBUILD does not support VSPROJs). In any case, my first installer project builds fine, the second one fails. If I reverse the order, same thing happends (i.e. the error does not follow the project). Looking at the output, I get the following errors:
Deserializing the project state for project '[MyProject].dbproj'
[MyProject].dbproj : error : Error HRESULT E_FAIL has been returned from a call to a COM component.
Also, I get:
Package 'Microsoft.VisualStudio.TestTools.TestCaseManagement.QualityToolsPackage, Microsoft.VisualStudio.QualityTools.TestCaseManagement, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' failed to load
It looks as though the first build tries to serialize the DB project (and it says it succeeds, but there is no DBML file anywhere). Then the second build tries to deserialize the DB project and fails.
I've tried resetting env settings (using the /resetusersettings flag) as well as using the /resetskippkgs flag. Nothing works.
Any ideas?
When you shell out to DevEnv, are you building that specific project (.vdproj file), or are you building the solution? It sounds like VS is trying to open the solution on the build machine and the database and test project systems aren't present.
Have you considered porting your setup project to WiX?
Start simple. Unless you're well versed in the problem you're trying to solve it's usually best to try it "by hand" before getting it running as part of a TFS build. RDP into the build server and try running the necessary commands at the command line and see what happens. You can even go simpler than that and RDP into the build machine and load Visual Studio and build it.
Bottom line is that if you can't get it to build within Visual Studio or at the command line by calling devenv.exe it won't work as part of the team build.
I am using the below Exec task to do precisely what you are doing as part of a TFS build. So I know this works. Your platform and configuration may vary depending on what you're building. The nice thing about this is that you'll have a log file at C:\Temp\MSIBuildOutputLog.txt that you can analyze for additional errors and information.
<Exec Command=""C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\devenv.exe" "$(PathToSolution)\solution.sln" /Build "Release|Mixed Platforms" /out "C:\Temp\MSIBuildOutputLog.txt"" />
One important thing to note... There is a bug in VS2010 which causes MSI generation to fail when you try to run it at the command line using devenv.exe. This took me days to find and figure out, but you need this hotfix. Worked like a charm...
http://archive.msdn.microsoft.com/KB2286556
Actually it's the deployment projects that don't support msbuild. FWIW, this is all deprecated in the next release of Visual Studio so you might want to start looking at InstallShield Limited Edition and/or Windows Installer XML now before spending too much time on dead end, broken technology. Both of these alternatives have proper MSBuild support aswell as many other improvements.
It would be perhaps better and quicker to adopt WIX (Windows Installer XML) which is the technology MS now recommends to use within VS/MSBuild/TFSBuild environment to crate MSIs.
It is relatively easy to setup and integrate within your VS Solutions. It uses XML based files to describe your MSIs and uses these files to create your MSIs when you compile.
I would start by downloading Wix from http://wix.codeplex.com/
Once installed you would be able to use the VS2010 integration of Wix based projects to create MSIs. To get started quickly simply add a new Wix project to your solution and reference the projects whose output you wish to combine into an MSI. Next you can run a tool called "Heat" which is included with Wix toolkit to generate the XML files by scanning your projects.
Once you have these XML files, add them to your Wix project and compile.
I have a solution containing my main project and a test project using NUnit. Everything compiles but when I run NUnit I get the exception below after the tests load, and the tests fail. I've added the main project as a reference, and I have $(ProjectDir)bin/Debug/$(TargetName)$(TargetExt) in the arguments for NUnit in the external tools setup, with a blank initial directory.
MyMainProjectTests.Database.TestAddDelete:
System.BadImageFormatException : Could not load file or assembly 'MyMainProject,
Version=1.1.1.0, Culture=neutral, PublicKeyToken=null' or one of its
dependencies. An attempt was made to load a program with an incorrect format.
TearDown : System.Reflection.TargetInvocationException : Exception has been
thrown by the target of an invocation.
----> System.BadImageFormatException : Could not load file or assembly
'ChickenPing, Version=1.1.1.0, Culture=neutral, PublicKeyToken=null' or one
of its dependencies. An attempt was made to load a program with an incorrect
format.
After scouring for hours the only thing I've found is a bug in VS2005 which mentions the /bin and /obj directories, but the answer provided didn't help.
Any solutions?
Instead of setting up NUnit as an External Tool, I set the unit test project as the StartUp project. In the project's Properties screen, set the Start Action to "Start external program" and point it to nunit.exe. In the Start Options section, I specify the test assembly (no path necessary) in the "Command line arguments" box. At this point, simply press F5 to start up NUnit.
Use the nunit-x86.exe instead of nunit.exe as your runner.
A better longer term solution may be to buy ReSharper that includes a much nicer test runner for NUnit that fully integrates into Visual Studio. It auto detects your .NET project type (x68 or x64). ReShaper comes with tons of other features of which unit testing is just one. Their test runner also integrates with their DotCover code coverage analyser.
You may find that you'll need a later version of Visual Studio to use ReSharper. The latest version works with Visual Studio 2013 Community Edition that you can get for free though I understand you may have issues upgrading some project features from such a rather old VS2005 project.
I don't have any affiliation with ReSharper.
Are you running on x64? You will get that error if loading a x64 bit from x86 and vise versa. Also, the path you are trying to create should be the $(TargetPath) macro.
Just set "Platform target" of Tests project to "x86".
Is your main project a .exe or a .dll? Older versions of .NET couldn't reference an .exe, so that might be the problem.
In either case, I'd expect problems if the main assembly didn't end up somewhere accessible by your test assembly (for example, in the same directory). You could check that, and if not make it so, perhaps by having Visual Studio copy the referenced (main) assembly to the local directory.
The "An attempt was made to load a program with an incorrect format." makes me wonder if the "missing assembly" theory is right, but without more info, it's the best guess I can think of.
Go the the NUnit install (example: C:\Program Files (x86)\NUnit 2.6.3\bin) location and open nunit-86.exe.